Hi, I'm trying lemmeknow and I gotta say that it works quite well with links (e.g. YouTube channels, wallets and so on), but it misses to detect the easiest things. For example it is unable to recognize a base64 encoded text.

Nice work though.

When opening the lemmeknow webpage with a URL such as this: https://swanandx.github.io/lemmeknow-frontend/?q=search+term It should use this as input and try to figure out what the search term could be.

Why?

Browser Search engines.

With this feature you could register lemmeknow as a search engine for your browser. You could (for example) use the alias lmk to search lemmeknow.

Then typing lmk dQw4w9WgXcQ would lead you to find out what the ID stands for.

Currently lemmeknow uses the fancy_regex crate for matching regex. The problem is that it doesn't support bytes. The regex crate, however does: https://docs.rs/regex/1.0.0/regex/bytes/index.html

If there is no reason to use fancy_regex then we should switch. Both pyWhat and lemmeknow only support ASCII strings. We need to support UTF-8, UTF-16, etc. and bytes.

See this equivalent pyWhat issue: https://github.com/bee-san/pyWhat/issues/34

This project uses same regex database as PyWhat , we need some performance benchmarks against it <3 !

• For identifying single text
• For analyzing strings from a file
• Calling function multiple times through API i.e. lemmeknow::what_is("text here") for lemmeknow and Identifier.identify("text") for pyWhat.

API documentations available here 😸 :-

• lemmeknow
• pyWhat

We have entries for Rarity and Tags in database. We need to implement a filter so that user can filter based on rarity and/or tags.

For example, lemmeknow --rarity 0.2:0.6 --tags Credentials TEXT.

making it a module will be nice idea 😄 src/output/filter.rs

Nix package is added in https://github.com/NixOS/nixpkgs/pull/194268 and I added myself as a maintainer in https://github.com/NixOS/nixpkgs/pull/196953.

We have Exploit for some identifications. It would be great if we could show them if user passed -v i.e. verbose flag on cli.

{
      "Name": "Mailchimp API Key",
      ...
      "Description": null,
      "Exploit": "Use the command below to verify that the API key is valid (substitute <dc> for your datacenter, i. e. us5):\n  $ curl --request GET --url 'https://<dc>.api.mailchimp.com/3.0/' --user 'anystring:API_KEY_HERE' --include\n",
      "Rarity": 0.8,
      "URL": null,
     ...
   },

The regex.json file also have Examples for some regexes,

{
      "Name": "Capture The Flag (CTF) Flag",
      "Regex": "(?i)^(flag\\{.*\\}|ctf\\{.*\\}|ctfa\\{.*\\})$",
      "plural_name": false,
      "Description": null,
      "Rarity": 1,
      "URL": null,
      "Tags": [
         "CTF Flag"
      ],
      "Examples": {
         "Valid": [
            "FLAG{hello}"
         ],
         "Invalid": []
      }
   },

We need to check if the regex is matching those examples correctly to validate it! For that, we can create a file under tests like lemmeknow/tests/validate_regexes.rs and just parse the JSON file and validate it there.

Add some more fields to Cargo.toml to optimize release builds more. Reference: https://nnethercote.github.io/perf-book/build-configuration.html

Also adds CI checks that run cargo check, cargo test, cargo fmt, and cargo clippy

Input: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c

Output: Found Possible Identifications :) | Matched text | Identified as | Description | |--------------------|--------------------|-----------------| | eyJhbGciOiJIUzI1..._adQssw5c | Litecoin (LTC) Wallet Address | URL: https://live.blockcypher.com/ltc/address/eyJhbGciOiJIUzI1..._adQssw5c |

Expected "Identified as" would be JWT. Shortened the JWT for better visibility.

Another example:

Input: eyJhbGciOiJIUzI1NiJ9.eyJmb28iOiJiYXIifQ.JTvQIxZOL_-00JdKfTAEmhV-a6KUlB6OUWM8NuN7MN8 Output: No Possible Identifications :(

Expected "Identified as" would be JWT.

I am debugging if my program is broken or if LemmeKnow is broken, there is no unit tests in LemmeKnow so I cannot prove it works. Please add unit tests for the API :)

    /// Only include the Data which have at least one of the specified `tags`
    pub tags: Vec<String>,
    /// Only include Data which doesn't have any of the `excluded_tags`
    pub exclude_tags: Vec<String>,

String is a growable buffer, we do not expect them to grow so we should use str

TryHackMe flag will be min 3 ( as it must have thm in it ) and there is no max limit.

YouTube Video ID will be 11 characters long, ( n92YrzELBJU )

We need a list with this for every regex.

• if there is no fixed length, put *
• if min but no max, use 3-*
• can only be 8 or 10, use 8/10
• exact length 11, use 11

Many regex identify text which have fixed size range, if we filter based on it first, we might optimize our algorithm. Suggestions are welcomed for other ways to optimize.

Following regex won't compile as regex crate doesn't support look-around.

If possible, can we rewrite them in such a way that they don't use look-around??

• [ ] Internet Protocol (IP) Address Version 6
• [ ] Bitcoin (₿) Wallet Address
• [ ] American Social Security Number
• [ ] Date of Birth
• [ ] JSON Web Token (JWT)
• [ ] Amazon Web Services Access Key
• [ ] Amazon Web Services Secret Access Key
• [ ] YouTube Video ID

Update them in src/data/regex.json

PS: You can run following command in repo if you want to see exact syntax error of regex ( or just use regex crate to compile them one by one )

cargo test validate_regex_examples -- --show-output