Implements #40.

Changes:

• Adds Node::classes which returns an iterator, and Node::id for convenience.
• Documents which CSS selectors are equivalent to which predicates.
• Renames Attr(s, ()) to HasAttr (breaking change; can be removed if desired).
• Adds Nothing predicate, the opposite of Any.
• Adds Id predicate, which calls Node::id.
• Adds Root predicate, that ensures that the node has no parents.
• Adds Empty predicate, which allows comment children to be compatible with CSS.
• Adds AttrMatches predicate for convenience; it takes a string for an attribute and a function to test the string.
• Adds AttrContains, AttrStartsWith, and AttrEndsWith which call the relevant string functions.
• Adds AttrContainsWord and AttrLang, which are kind of esoteric but are equivalent to the [attr~=word] and [attr|=lang] selectors.

Once all of the CSS selectors are added we can potentially allow parsing &str -> Box<Predicate>.

Currently, one cannot create a Document from a String, as is the case when reading a response from Hyper into a String. With From, this is now possible.

Example of new allowed behaviour:

let client = hyper::Client::new();
let mut response = try!(client.get(&request_url).send());
let mut body = String::new();
try!(response.read_to_string(&mut body));

let dom = Document::from(body);

Getting the last element leads to some non-trivial lifetime wrangling when implemented using iterators over a Selection, so adding a last() method helps avoid those difficulties.

Fixes #17

There haven't been any commits since over a year and I'm wondering whether @utkarshkukreti is still developing this. @utkarshkukreti, if you're reading this, do you plan to keep developing/maintaining this library?

Is it possible to access attributes using select.rs without knowing their names beforehand? If not, could an interface for iterating over (attr name, attr value) pairs be added?

Hi @utkarshkukreti

We want to show your excellent crate in rust-cookbook https://github.com/brson/rust-cookbook/pull/183 :smile: but due to bug in our documentation test driver we need for all of intermediate dependencies to be on the same level.

Long story (really!) made short. We would require to update select to latest html5ever released today.

If there is anything wrong with the PR I will gladly fix it!

Best Regards

This requires using the newly split-off RcDom implementation from html5ever.

Since this is now clearly marked that it shouldn't be used in production, this raises the question, should a production-ready DOM implementation be used instead for select.rs?

Looks like example outdated because I couldn't compile it:

src/main.rs:16:52: 16:62 error: no method named `descendant` found for type `select::predicate::Attr<&str, &str>` in the current scope
src/main.rs:16     for node in document.find(Attr("id", "hmenus").descendant(Name("a"))) {
                                                                  ^~~~~~~~~~
src/main.rs:22:58: 22:62 error: no method named `take` found for type `select::selection::Selection<'_>` in the current scope
src/main.rs:22     for node in document.find(Class("question-summary")).take(5) {
                                                                        ^~~~
src/main.rs:22:58: 22:62 note: the method `take` exists but the following trait bounds were not satisfied: `select::selection::Selection<'_> : std::iter::Iterator`
src/main.rs:25:49: 25:59 error: no method named `descendant` found for type `select::predicate::Class<&str>` in the current scope
src/main.rs:25         let answers = node.find(Class("status").descendant(Name("strong")))
                                                               ^~~~~~~~~~
src/main.rs:31:53: 31:63 error: no method named `descendant` found for type `select::predicate::Class<&str>` in the current scope
src/main.rs:31         let asker = node.find(Class("user-details").descendant(Name("a")))
                                                                   ^~~~~~~~~~
src/main.rs:49:10: 49:16 error: no method named `unwrap` found for type `select::selection::Selection<'_>` in the current scope
src/main.rs:49         .unwrap()
                        ^~~~~~
error: aborting due to 5 previous errors

$ cargo --version
cargo 0.12.0-nightly (6b98d1f 2016-07-04)
$ rustc --version
rustc 1.11.0 (9b21dcd6a 2016-08-15)

As mentioned in https://github.com/utkarshkukreti/select.rs/issues/61, I find myself in need of a Debug implementation for Find.

The only member of Find that does not necessarily implement Debug is predicate (which may be a closure), so I left it out from the Debug impl. This is in line with Rust's Filter (and other iterators).

The following code cause panic:

use std::fs::File;
use std::io::prelude::*;

#[macro_use]
extern crate error_chain;

use select::document::Document;

error_chain! {
    foreign_links {
        Reqwest(reqwest::Error);
        Std(std::io::Error);
    }
}

fn main() -> Result<()> {
    /*
     * This download the file
     *
     */
    // let html = reqwest::get("http://sampsonsheriff.com/")?.text()?;
    // let mut file = File::create("src/panic.txt")?;
    // file.write_all(html.as_bytes())?;


    let document = Document::from(include_str!("panic.txt"));

    Ok(())
}

Here is the stack trace:

thread 'main' panicked at 'assertion failed: c.is_some()', /home/abdullah/.cargo/registry/src/github.com-1ecc6299db9ec823/html5ever-0.18.0/src/tokenizer/mod.rs:555:9
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
   1: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:70
   2: std::panicking::default_hook::{{closure}}
             at src/libstd/sys_common/backtrace.rs:58
             at src/libstd/panicking.rs:200
   3: std::panicking::default_hook
             at src/libstd/panicking.rs:215
   4: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:478
   5: std::panicking::begin_panic
             at /rustc/2aa4c46cfdd726e97360c2734835aa3515e8c858/src/libstd/panicking.rs:412
   6: <html5ever::tokenizer::Tokenizer<Sink>>::discard_char
             at /home/abdullah/.cargo/registry/src/github.com-1ecc6299db9ec823/select-0.4.2/<::std::macros::panic macros>:3
   7: <html5ever::tokenizer::Tokenizer<Sink>>::step
             at /home/abdullah/.cargo/registry/src/github.com-1ecc6299db9ec823/html5ever-0.18.0/src/tokenizer/mod.rs:570
   8: <html5ever::tokenizer::Tokenizer<Sink>>::run
             at /home/abdullah/.cargo/registry/src/github.com-1ecc6299db9ec823/html5ever-0.18.0/src/tokenizer/mod.rs:362
   9: <html5ever::tokenizer::Tokenizer<Sink>>::feed
             at /home/abdullah/.cargo/registry/src/github.com-1ecc6299db9ec823/html5ever-0.18.0/src/tokenizer/mod.rs:220
  10: <html5ever::driver::Parser<Sink> as tendril::stream::TendrilSink<tendril::fmt::UTF8>>::process
             at /home/abdullah/.cargo/registry/src/github.com-1ecc6299db9ec823/html5ever-0.18.0/src/driver.rs:88
  11: tendril::stream::TendrilSink::one
             at /home/abdullah/.cargo/registry/src/github.com-1ecc6299db9ec823/tendril-0.3.1/src/stream.rs:47
  12: <select::document::Document as core::convert::From<tendril::tendril::Tendril<tendril::fmt::UTF8>>>::from
             at /home/abdullah/.cargo/registry/src/github.com-1ecc6299db9ec823/select-0.4.2/src/document.rs:53
  13: <select::document::Document as core::convert::From<&'a str>>::from
             at /home/abdullah/.cargo/registry/src/github.com-1ecc6299db9ec823/select-0.4.2/src/document.rs:133
  14: reqwest_test::main
             at src/main.rs:26
  15: std::rt::lang_start::{{closure}}
             at /rustc/2aa4c46cfdd726e97360c2734835aa3515e8c858/src/libstd/rt.rs:64
  16: std::panicking::try::do_call
             at src/libstd/rt.rs:49
             at src/libstd/panicking.rs:297
  17: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:92
  18: std::rt::lang_start_internal
             at src/libstd/panicking.rs:276
             at src/libstd/panic.rs:388
             at src/libstd/rt.rs:48
  19: std::rt::lang_start
             at /rustc/2aa4c46cfdd726e97360c2734835aa3515e8c858/src/libstd/rt.rs:64
  20: main
  21: __libc_start_main
  22: _start

I have attached the file in case the website change. panic.txt

Added example of how to just get the plain text from a webpage. (E.g. for machine learning sometimes you just want access to plain text).

If there's a neater way to do it, I'd love to see it - I couldn't find a specific root node function on document? Did I miss one, if not it would be nice to call it out as a specific function rather than nth(0).

https://github.com/time-rs/time/issues/293

https://github.com/utkarshkukreti/select.rs/blob/master/Cargo.toml#L14

https://github.com/servo/html5ever/blob/master/xml5ever/Cargo.toml

It appears that bumping the html5ever dependency would resolve it, but I haven't checked myself

I'm using select in a broad web crawler, everything is working fine until it does not... Web is full of surprises and magic input nobody expects to find that can crash almost anything(just by the sheer amount of data web contains)... sometimes, sometimes select will go into deep recursion that causes stack overflow(and I have thread stack size set to 32mb), now this might be a bug in select which can be fixed...(but I cannot even log it... because need to run crawler under gdb to catch such stuff it seems)

What I'm looking for is a way to limit the depth of recursion in select's code to ensure we do not stack overflow. Afaik stack overflows panics are impossible to recover in rust and lead to forceful program exit :\

Hi,

What is next() doing? in the below code? And is there documentation or something about how to use the next()? Thanks!

    for node in document.find(Class("question-summary")).take(5) {
        let question = node.find(Class("question-hyperlink")).next().unwrap();

Hi, I'v encounter STATUS_STACK_OVERFLOW when load a big xml file, abount 341KB. On win7 64bit, vs2013.

fn main() {
    use select::document::Document;
    use select::predicate::{Name, Predicate};

    let document = Document::from(
        include_str!("8144940")
    );
    `println!("Hello,` world!");
}

8144940.zip