Hi @maxymkuz and @LastGenius-edu,

Please take a look at the next Dockerfile snippet.

FROM ekidd/rust-musl-builder:latest AS builder

RUN mkdir -p /home/rust/my-project
WORKDIR /home/rust/my-project
VOLUME ["/home/rust/myproject"]

# cache dependency artifacts. 'Cargo build' builds 
# both dependencies and project files, 
# splitting the project into dependencies with single 
# empty main and current project sources.
COPY --chown=rust:rust Cargo.toml Cargo.toml
COPY --chown=rust:rust Cargo.lock Cargo.lock
RUN echo 'fn main() {}' > src/main.rs
RUN cargo build --release

# build the project using prebuilt deps
RUN rm /home/rust/my-project/src/main.rs
COPY --chown=rust:rust ./src ./src
RUN cargo build --release

This trick solves the issue with dependencies caching during cargo build. It splits the build process into two steps: first builds dependencies with an empty main and commits the separate image layer. Docker will reuse this layer later during the project build. .`

This is just a prototype that needs to be implemented for each Rust project. There is no need to use a separate project; you can implement it for /home/rust/src.

We've completely transferred to Elasticsearch, improved text parsing and set up a proper database backend. And search now works!

• [x] Set up Elasticsearch and transfer existing PostgreSQL functionality to it
• [x] Implement database backend to handle requests from crawlers and web backend
• [x] Set up web backend to properly communicate with the database backend
• [x] Connect the Python crawler to the database.
• [x] Improve Docker infrastructure (still a lot to work on)

Worked a little more on the crawler, and now it uses a thread pool where each thread does asynchronous I/O with given webpages.

Updating the list from the previous Pull Request #1, tasks done for this week are:

Completed:

• [x] A minimal Docker image for the crawler.
• [x] A CLI app that takes a text input file and then crawls through these webpages in parallel, collecting more and more links.
• [x] The app saves the collected structured data in plain form into an output file.
• [x] Implement into asynchronous implementations for each worker in the thread pool
• [x] Improve link collection, stop considering anchors on the same page as a different page
• [x] Add more documentation and split the file into several modules

Things left to do, roughly in the descending order of importance:

• [ ] Add tests for the crawler and thread pool implementations
• [ ] Test the module dynamically and figure out the best amount of threads and simultaneous async requests
• [ ] Fix up progress display in a Docker image
• [ ] Add more detailed progress info on what each thread is doing

I've implemented a simple crawler that uses thread pool and message passing to crawl webpages in parallel. While it uses most of CPU's resources and does everything we agreed upon for this week, there is still a lot of room for improvement, more on this below.

Completed:

• [x] A minimal Docker image for the crawler.
• [x] A CLI app that takes a text input file and then crawls through these webpages in parallel, collecting more and more links.
• [x] The app saves the collected structured data in plain form into an output file.

Things left to do, roughly in the descending order of importance:

• [ ] Look into asynchronous implementations for each worker in the thread pool
• [ ] Improve link collection, stop considering anchors on the same page as a different page
• [ ] Add more documentation and split the file into several modules
• [ ] Add tests for the crawler and thread pool implementations
• [ ] Fix up progress display in a Docker image