This pull request includes several improvements to the Websurfx project's documentation. The changes were made to improve the clarity and completeness of the README and CONTRIBUTING files, and to ensure that contributors have a clear understanding of the project's goals and requirements.

The changes include:

• Refactoring the project goals for clarity and improved grammar.

• Adding a brief introduction to the project and its main features in the README.

• Clarifying the installation process with specific commands and instructions for the rolling and master branches in the README.

• Expanding the contributing guidelines to include required knowledge and guidelines in the README.

• Updating the CONTRIBUTING file to improve its clarity and organization, reorganizing the existing guidelines and clarifying the requirements for contributing to the project.

These changes should improve the readability and usability of the project's documentation, making it easier for users and contributors to understand and use the project effectively.

What does this PR do?

This PR adds more request headers to spoof the requests and better evade being flagged as a bot.

Why is this change important?

This change is necessary to not get flagged and IP banned while scraping and aggregating results from upstream search engines.

Author's checklist

• [x] add appropriate cookie header
• [x] fix Content-Type header if not correct.

Related issues

Closes #8

What does this PR do?

This PR optimizes code by adding the ability to fetch results from upstream search engines concurrently using tokio::join!().

Why is this change important?

This allows for fetching and aggregating results faster and makes it more available for large scale server use.

Author's checklist

• [x] optimize code by adding the ability to fetch code concurrently.
• [x] add random delays between each 2 concurrent requests.

Related issues

Closes #7

Need to write different documentation pages about theming, installation, configuration, etc in the wiki and also make empty links in the project to point to their respective sections int the wiki

What does this PR do?

Improves readme.md and makes it look more attractive

Why is this change important?

This PR is important as this improves the readme which helps others visiting the project better understand the project and its goals.

Author's checklist

• [x] Fix formatting errors
• [x] Add images
• [x] Add status badges
• [x] Add icons
• [x] Add more sections

Related issues

Closes #9 and Closes #3

Need to rewrite and reformat readme to look more attractive with icons, images and gifs and with styling like putting images in center.

Example projects with awesome readmes:

• https://github.com/Lissy93/dashy/blob/master/README.md
• https://github.com/vlang/v/blob/master/README.md

The code needs some extra improvements like adding cookie header with appropriate cookies for each upstream search engine and also adding other headers to better spoof and protect against bot detection and better emulate human behavior and also evade IP blocking/banning.

Files seeking changes are:

• src/engines/duckduckgo.rs
• src/engines/searx.rs

Optimise code by using tokio async to aggregate search results from each upstream server concurrently/simultaneously with random delay in between each simultaneous requests.

For example: If there are 4 threads than four simultaneous requests could be made to each server and fetch their result scrape them and aggregate their results and then add a random delay and continue with another 4 and so and so forth.

Example code:

use std::error::Error;
use tokio::time::delay_for;
use reqwest::Client;

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let client = Client::new();
    let urls = vec![
        "https://example.com/page1",
        "https://example.com/page2",
        "https://example.com/page3",
    ];

    let mut tasks = Vec::new();
    for url in urls {
        tasks.push(tokio::spawn(async move {
            let resp = client.get(url).send().await?;
            let body = resp.text().await?;
            // Process the HTML body here
            Ok(())
        }));
    }

    for task in tasks {
        task.await??;
    }

    Ok(())
}

Currently the readme (which is written in markdown) does not have any status badges but the readme should clearly show a few status badges these are:

• Build status (showing the github workflow status)
• Issue and pull request trackers (showing the number of open issues and pull requests )
• License (showing which license is the project using)
• Rating (showing the number of stars for this project on github)
• Repo size (showing the size of this repository)

These all build statuses can be added using the shields.io website.

In markdown file html code can be added something like this:

<a href="https://github.com/neon-mmd/websurfx/graphs/contributors">
   <img src="https://contrib.rocks/image?repo=neon-mmd/websurfx" />
</a>

Note: All contributions should go into the rolling branch of this repository and for before you contribute fork this project add your appropriate changes to your fork and then add a PR mentioning this issue.

At present the about page of the website just contains dummy text and it needs to show a clear and concise description about the project in the website. The about page's html code (which uses handlebars templating) is located under the public/templates/ folder in the project and its css stylesheet is located under public/static/themes folder.

Note: All contributions should go into the rolling branch of this repository and for before you contribute fork this project add your appropriate changes to your fork and then add a PR mentioning this issue.

We have 9 colorschemes for the website theme located under the public/static/colorschemes/ folder in the project. These colorschemes allows the user to change the color of the theme being used in the website. They all are written in css and have 9 variables in each colorscheme file to play with. At present all colorschemes except the catppuccin-mocha needs to be fixed and also you can add more famous colorschemes like tokyo-night into the project.

Notes:

1. You can refer to the simple theme located under the public/static/themes/ folder to know where each color variables are being used.
2. All contributions should go into the rolling branch of this repository and for before you contribute fork this project add your appropriate changes to your fork and then add a PR mentioning this issue.