Based on: https://github.com/brave/adblock-lists/issues/355 request. The most comprehensive list and also used by uBO as the default Japanese filter.

Since ABP Japanese is not well supported and has many problematic filters; This change will help fix a few work arounds we needed to make (which will need to be reverted)

https://github.com/brave/adblock-lists/pull/366 https://github.com/brave/adblock-lists/pull/319 https://github.com/brave/adblock-lists/pull/274 https://github.com/brave/adblock-lists/pull/270

More Japanese filter issues, https://github.com/brave/adblock-lists/pulls?q=is%3Apr+is%3Aclosed+label%3Aregional

In light of recent developments that gave me a pretty good list conversion script, and closer connections to Fanboy (who seem to occasionally contribute to you guys), I will hereby do this new attempt at getting my Nordic list added to Brave Browser's adblocker.

Some of the strings I replaced with "xxxxxxxxxx", as it was not obvious to me how to fill them with correct information when they're edited with GitHub's web GUI. You should almost guaranteed have been given full rights to do changes of your own to my PR branch.

I've tried to share an instance of Engine across threads and encountered that it can not be shared across threads as it is not thread-safe. Can you please replace Rc and RefCell with Arc and RwLock/Mutex so we can use this library in threads? Thanks!

Since Brave's adblocker announced it's compatible with UbO's filters, I used RuAdList-uBO.txt only to realize it doesn't support a filter including other filters like that. I ended up have to add each filter file of the ruadlist collection manually (so many of them!). It's somewhat worked, but still not as optimal as it is on UbO. On news.mail.ru there is still empty ad container boxes. These boxes should be removed (as they are on UbO). I'm always using dark mode, having a white box on a dark background make my eyes hurt. Please fix it.

Update: my guess is Brave's adblocker does not fully support all of UbO's features so filter lists like css-fixes-experimental.txt and js-fixes-experimental.txt doesn't work optimally, I might have to install Stylus and ViolenMonkey to use the corresponding fixes like when using with Adblock Plus as stated on the easylist forum. But, it shouldn't be the case! Brave's adblocker should be compatible with all UbO filters isn't it?

There was an issue in EasyList recently that broke some ad blockers. Even though the report doesn't mention Brave, the issue should have affected Brave too. Meanwhile, uBlock Origin was barely affected because it interpreted the filter in question as a hostname.

Now this raises the question as to how such filters should be interpreted in general.

The Brave implementation has two modes: standard and hosts. In hosts mode, a hostname on a line by itself is interpreted as a hostname. In standard mode, the same hostname must be specified using Adblock Plus syntax. For example, example.com must be encoded as ||example.com^.

Should Brave adopt the uBlock Origin style in standard mode?

The crux of the issue is that Adblock Plus-style filters tend to overblock by default. A filter like ad.png would block not only the URL https://example.com/ad.png (intended) but also the URL https://example.com/ipad.png (not intended). Similarly, the filter example.com would block https://example.com/ad.png (intended) but would also block https://example.computer/ad.png (not intended).

Whereas filter developers are aware of such nuances, and indeed there are no such filters in EasyList et al, the average user who types a filter into a box (https://github.com/brave/brave-browser/issues/8838) is unlikely to go look up the syntax, learn it, and then correctly specify the hostname example.com as ||example.com^. In other words, the Adblock Plus syntax is not exactly user-friendly. Based on the recent incident, it also appears to be a bit of a footgun. We don't know how many end users are experiencing broken web pages because they were savvy enough to write their own filters by hand.

The principle of least surprise dictates that if the pattern in a URL filter looks like a hostname, it should be interpreted as a hostname.

$ grep -Ei '^(@@)?[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)*(\$.*)?$' easylist.txt easyprivacy.txt | wc -l
0

This new "shorthand" syntax for hostnames would break 0 existing filters in EasyList and EasyPrivacy.

$ grep -Ei '^(@@)?\|\|[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)*\^(\$.*)?$' easylist.txt easyprivacy.txt | wc -l
31825

On the other hand, there are nearly 32,000 filters in EasyList and EasyPrivacy, out of a combined total of about 81,000 filters, that could be converted to use the new syntax at some point, reducing the size of the raw filter text by ~100 KiB.

Not only would the shorthand syntax be more user-friendly, it might also have bandwidth, memory, and disk usage benefits over time.

WDYT?

Rules with statically typed attributes don't seem to be handled by the library. For instance, div[class='a-class'] is equivalent to div.a-class. However, in the code below and unless I am mistaking, the "hidden_class_id_selectors" method should return a non empty vector corresponding to the first rule in the set.

The "hidden_class_id_selectors" seems to be using the masks set in the CosmeticFilter object related to the rule ("div[class='a-class']") , however, printing it shows no masks being set for it, which causes the CosmeticFilterCache to store the rule in misc_generic_selectors which is in turn ignored by the method "hidden_class_id_selectors".

let rules = [
            String::from("##div[class='a-class']"),
            String::from("##div[class*=\"Ads\"]"),
            String::from("##.a-class")
        ];
// For debugging purpose
rules.iter().for_each(|rule| {
    let cf: CosmeticFilter = CosmeticFilter::parse(rule, false).unwrap();
    println!("{:?}", cf);
});
//----

let blocker = Engine::from_rules_debug(&rules, FilterFormat::Standard)
let example_class_report = blocker.hidden_class_id_selectors(&["div.a-class".into()], &[], &HashSet::default());

println!("Report for a-class: {:?}", example_class_report);

Are these rules supported in the current version ? If yes, how are they supposed to be used ? Similarly, are composed rules (for example, "div#ads > .advertisement") also handled ? Some additional documentation/examples should be added regarding the usage of cosmetic rules.

Please let me know if my facts are wrong as I am getting started with the Rust language/library.

Since Prebake Cookie hasn't been updated since 2018, we should replace this with Easylist Cookie.

Should help with cookie notices and resolve brave/brave-browser#5318 Will be more affective when we roll out cosmetic filtering.

Rebase of https://github.com/brave/adblock-rust/pull/67 (just had some merge issues)

Also re-did component_id: and base64_public_key:

PEM's also updated/uploaded.

Bumps tokio from 1.8.0 to 1.8.4.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

This will block Notification servers being used in Brave;

From our recent merge of Notification lists; https://github.com/brave/adblock-lists/pull/297

Any questions let me know.

On a fresh clone of v0.5.6 I encountered a few compilation errors. After this PR, the project should compile without errors or warnings using RUSTFLAGS="-D warnings" cargo check --workspace --all-targets --all-features, and that command has been added to CI to prevent compilation errors from hiding behind feature flags in the future.

All tests pass except for live.rs tests, which require a BRAVE_SERVICE_KEY that isn't checked into the repo. I managed to find that key, which does not appear to be a secret. However, I want to understand the purpose of the BraveServiceKey header before I fix those tests.

The changes are organized into three commits:

1. Swapped native-tls for rustls-tls in the reqwest dev dependency, which should be less sensitive to version changes in OS-specific openssl libraries. (I use Ubuntu 22.04, which uses libssl3 instead of libssl1.1, so I run into this problem frequently.)
2. Added CI step to keep main branch free of warnings & errors
3. Removed feature flag full-domain-matching which since 2019 would cause the build to fail. I also moved & rewrote an old failing test from tests/hashing_test.rs to src/filters/network.rs.

resolve #225

Add a new filter option to adblock called redirect-url (similar to redirect=) that allows loading of replacement resources via a URL instead of having to bundle all replacement resources in the client.

Bumps tokio from 1.21.1 to 1.23.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Used in uBO, https://github.com/uBlockOrigin/uAssets/blob/master/filters/badware.txt#L881 https://github.com/uBlockOrigin/uAssets/blob/master/filters/filters.txt#L19296

The match-case regex filters used for revolving adserver domains. From what I see, they are ignored in Brave

badware.txt:/^https:\/\/[0-9a-z]{3,}\.[-a-z]{10,}\.(?:li[fv]e|top|xyz)\/[a-z]{8}\/\?utm_campaign=\w{40,}/$doc,match-case,domain=life|live|top|xyz
badware.txt:/^https?:\/\/((?!www)[a-z]{3,}|\d{2})?\.?[-0-9a-z]{6,}\.[a-z]{2,6}\/(?:[a-z]{6,8}\/)?\/?\?u=[0-9a-z]{7}&o=[0-9a-z]{7}/$doc,frame,match-case,domain=buzz|com|de|fun|guru|info|life|live|mobi|online|pw|site|space|top|us|xyz
badware.txt:/^https?:\/\/((?!www)[a-z]{3,}|\d{2})?\.?[-0-9a-z]{6,}\.[a-z]{2,6}\/(?:[a-z]{6,8}\/)?\/?\?o=[0-9a-z]{7}&u=[0-9a-z]{7}/$doc,frame,match-case,domain=buzz|com|de|fun|guru|info|life|live|mobi|online|pw|site|space|top|us|xyz
badware.txt:/^https:\/\/(?:www\d\.)?[-a-z]{6,}\.(?:com|info|net|org)\/(?=[-_a-zA-Z]{0,42}\d)(?=[-_0-9a-z]{0,42}[A-Z])[-_0-9a-zA-Z]{43}\/\?cid=[-_0-9a-zA-Z]{16,36}(?:&qs\d=\S+)?&sid=[_0-9a-f]{1,32}$/$doc,match-case,domain=com|info|net|org
badware.txt:/^https:\/\/(?:www\d\.)?[-a-z]{6,}\.(?:com|info|net|org)\/(?=[-_a-zA-Z]{0,42}\d)(?=[-_0-9a-z]{0,42}[A-Z])[-_0-9a-zA-Z]{43}\/\?sid=[_0-9a-f]{1,32}(?:&qs\d=\S+)?&cid=[-_0-9a-zA-Z]{16,36}$/$doc,match-case,domain=com|info|net|org
badware.txt:/^http:\/\/[a-z]{5}\.[a-z]{5}\.com\/[a-z]{10}\.apk$/$doc,match-case,domain=com
filters.txt:/\/[A-Z]\/[-0-9a-z]{5,}\.com\/(?:[0-9a-f]{2}\/){3}[0-9a-f]{32}\.js$/$script,1p,match-case
filters.txt:/^https?:\/\/(?:[a-z]{2}\.)?[0-9a-z]{7,16}\.com\/[a-z](?=[a-z]{0,25}[0-9A-Z])[0-9a-zA-Z]{3,26}\/(?:[1-5]\d{4}|[3-9]\d{3})\??(?:_=\d+|v=\d)?$/$frame,script,xhr,popup,3p,match-case
filters.txt:/^https?:\/\/(?:[a-z]{2}\.)?[0-9a-z]{7,16}\.website\/[a-z](?=[a-z]{0,25}[0-9A-Z])[0-9a-zA-Z]{3,26}\/(?:[1-5]\d{4}|[3-9]\d{3})\??(?:_=\d+|v=\d)?$/$frame,script,xhr,popup,3p,match-case
filters.txt:/^https?:\/\/[a-z]{8,15}\.top(\/(?:\d{1,5}|0NaN|articles?|browse|index|movie|news|pages?|static|view|web|wiki)){1,4}(?:\.html|\/)$/$frame,3p,match-case
filters.txt:/^https?:\/\/[a-z]{8,15}\.top\/[a-z]{4,}\.json$/$xhr,3p,match-case
filters.txt:/^https?:\/\/[a-z]{8,15}\.top\/[-a-z]{4,}\.css\?aHR0c[\/0-9a-zA-Z]{33,}=?=?$/$css,3p,match-case
filters.txt:/^https?:\/\/[a-z]{8,15}\.top\/[a-z]{4,}\.png\?aHR0c[\/0-9a-zA-Z]{33,}=?=?$/$image,3p,match-case
filters.txt:/^https?:\/\/[a-z]{8,15}\.xyz(\/(?:\d{1,5}|0NaN|articles?|browse|index|movie|news|pages?|static|view|web|wiki)){1,4}(?:\.html|\/)$/$frame,3p,match-case
filters-2022.txt:/^https?:\/\/cdn\.[a-z]{4,6}\.xyz\/app\.js$/$script,3p,match-case
filters-2022.txt:/^https:\/\/a\.[-0-9a-z]{4,16}\.(?:club|com?|cyou|info|net|ru|site|top?|xxx|xyz)\/(?=[a-z]{0,6}[0-9A-Z])[0-9a-zA-Z]{7}\.js$/$script,match-case
privacy.txt:/^https:\/\/cdn\.jsdelivr\.net\/npm\/[-a-z_]{4,22}@latest\/dist\/script\.min\.js$/$script,3p,match-case
privacy.txt:/^https?:\/\/[-.0-9a-z]+\/script\.js$/$script,1p,strict3p,match-case

For https://github.com/brave/brave-browser/issues/26892

The idea is to use global regular expression storage instead of individual ones. This allows to challenge the long-term regexp memory consumption problem in the next PRs. A new RegexManager is in charge of matching/spawning/deleting regex instances. NetworkFilter is enforced to match using RegexManager.

Plans:

• track the last regex usage and drop the instance it after a few time;
• use less effective, but more memory-saving algorithms for some of the reg exps until it becomes a hot path;

Note:

• Thread safety is dropped, but could be restored via adding Mutex for RegexManager under the flag (do we really need it?).
• Using a standard hasher for u64 instead of AHasher results in 3% performance drop.. Not sure AHasher is the best option. I am open to other suggestions.
• Replacing a standard hasher in other places could also give a perf boost, worst trying.
• get_debug_info() method is supposed to be exposed to something like brave://adblock-internals.

I noticed this problem which shouldn't be crucial for most people but it would be nice if it gets fixed eventually, it applies to both, Custom filters and Internal/Default Lists.

But, I use uBo-Scriptlets and they have some nice scriptlets. When I was testing the ones to add Local storage items in Brave, I noticed that I can't get Brave to use the quotes symbol in a string.

This issue seems to only apply to backslash, and single and double quotes, so it is like the adblocker just decided to ignore any quote or double backslash (to add one backslash) in the rules.

Since many Local Storage items are JSON, they would need the quotes to work properly.

For example if I add the scritplet for setLocalItem to Brave and then use brave.com##+js(sli, test, {\"m-tcon\":true\,\"m-other\":true\,\"m-owl\":true\,\"m-game\":true\,\"m-ffz\":true\,\"m-addon\":true}) the quotes get ignored even if technically it should just work and don't act as normal quotes.

Using the backslash with the commas works as expected though, since the backslash is needed so the commas don't act as individual arguments for the scriptlet which would crash the browser if too many are in the rule.

So, it is not a huge issue and my workaround was to modify the scriptlet and use replace() and then it replaces the | to quotes symbol, so it is easy, but it would be nice if it just worked just by adding the backslashes to quotes (or without using backslashes for these symbols like uBlock does it automatically) instead of modifying the scriptlet.

Thank you.

I don't know if pre-processing directories are supported or not, https://github.com/brave-experiments/ad-block/issues/114 is archived without any notion. In particular, if-condition to speficy Brave Shield (env_brave or something) will be useful to list maintainer. Sure, Brave has its own unbreak list, but if a Brave-specific issue like https://github.com/brave/adblock-lists/issues/614 can be fixed in upstream lists by specifying Brave, this will save time of both Brave list maintainer (@ryanbr ) and me (or whoever filter author who mind Brave). Moreover, it will encourage more people to create and maintain lists for Brave. I believe if you can implement this, Adguard (@ameshkov) and uBlock Origin (@gorhill ) can add the same directive whose value will be false on them (this is completely different from https://github.com/uBlockOrigin/uBlock-issues/issues/1213).