~Tests fail when reading header from countries.fgb with error Unaligned { position: 84, unaligned_type: "f64", error_trace: ErrorTrace([TableField { field_name: "envelope", position: 48 }]) }~

~Reading features emit error Unaligned { position: 444, unaligned_type: "f64", error_trace: ErrorTrace([TableField { field_name: "xy", position: 436 }, VectorElement { index: 0, position: 92 }, TableField { field_name: "parts", position: 84 }, TableField { field_name: "geometry", position: 20 }]) }~

Edit: Works with flatbuffers 0.8.3. Waiting for FlatBuffers 2.0 release (https://github.com/google/flatbuffers/issues/6353).

As discussed in https://lists.osgeo.org/pipermail/proj/2019-June/008657.html it would be good if any future proof format of spatial features can explicitly represent temporal dimension i.e 4D.

By providing well known meta for possible dimensions X,Y,Z,M and T for a coordinate we can support 4D.

Open questions:

• Does unix timestamp work as temporal dimension?
• Can X,Y vs Z possibly/reasonably need separate temporal dimensions?

Regressed after 3.17.4. Error message with 3.18.0:

Module not found: Error: Package path ./lib/mjs/geojson is not exported from package /tmp/broccoli-1848999uiU6yCVIbUjp/cache-615-webpack_bundler_ember_auto_import_webpack/node_modules/flatgeobuf (see exports field in /tmp/broccoli-1848999uiU6yCVIbUjp/cache-615-webpack_bundler_ember_auto_import_webpack/node_modules/flatgeobuf/package.json)

• [ ] ~~Include examples which run off the local build (i'm tired of having to always modify them to point to my local 😆)~~ the naive implementation proved controversial, so not doing this for now.
• [x] make it easier to discover all the other examples - e.g. a meta-index which links to all of the examples.
  • full file
  • search
  • for both leaflet and open layers

Also, as the JS i/o gets better (#50), it might be neat to post slightly different examples:

• [x] search a very large file - I have such a file. Would you be interested in moving it to the fgb s3 bucket?
• [x] dynamic search (e.g. drag a bounding rect around)

Hi there, thank you for making such a great lib! I'm pretty new to both rust and flatgeobuf, and facing an issue when opening a fgb dataset with CRS EPSG:25832. It complains with:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Geometry("Utf8 error for string in 196..1969: invalid utf-8 sequence of 1 bytes from index 1542\n\twhile verifying table field `wkt` at position 188\n\twhile verifying table field `crs` at position 60\n")'

The dataset is a subset of Bavaria cycling network from https://www.ldbv.bayern.de/produkte/weitere/opendata.html , selected and exported to fgb format with QGIS. I am able to open it in QGIS, but met the above issue when reading it with flatgeobuf lib. For other dataset with EPSG:4326 system, there is no such problem. But I am not sure if the CRS is the reason. Could you help shed some light? Thanks a lot! For some reason, I could not upload the sample dataset here. sorry about that!

I see that your Leaflet/OpenLayers examples load the built source in the script tag, from the /dist/ folder

https://cdn.jsdelivr.net/npm/[email protected]/dist/flatgeobuf-geojson.min.js
https://cdn.jsdelivr.net/npm/[email protected]/dist/flatgeobuf-ol.min.js

However, there's currently no exported object from the module. So if I try to require the package, I get an error:

require('flatgeobuf')
// Error: Cannot find module 'flatgeobuf'

This means that it's not possible now (I think) to use it with a bundler, like Webpack or Browserify.

It would be helpful to export some objects, to be able to do one of

const flatgeobuf = require('flatgeobuf');
import flatgeobuf from "flatgeobuf";

flatgeobuf.deserialize(buffer)

Hi there,

I was having a play around either requesting flatgeobuf portions (eg by bbox) in the browser and I noticed that the slow part of the code was in the deserialize operation, compared to the data fetching which seemed to be really quick, that kinda surprised me.

So I had a poke through the code and noticed lots of use of .map functions. .map functions are generally considered very slow compared to regular for loops (there are stack of blog posts about this eg here). Obviously .map functions are nice in terms of readability, but if you're looking to get some performance gains out of the parsing then replacing them night be a good start, particularly where they are used on large arrays like coords.

There are also some other candidates I spotted on my quick scan

• Avoiding .slice on float arrays (this might be a bottleneck if it's being called for every coord)
• Removing the use of spread operators for concatenating arrays (although I don't think the use of this is wide spread)

I'd attempt the PR with a benchmark but TS drives me a bit bonkers sorry :)

Anyway, aside from that, I really the appreciate the project, from my initial play it's been very simple to use 👍

Thanks, Rowan

While having a deeper look at flatgeobuf, I started a Rust port. Don't know how much time I can invest, but flatbuf generated code looks already good so far.

When I import in this way: const geojson = require('flatgeobuf/lib/cjs/flatgeobuf.js');

it throw error:

node:internal/modules/cjs/loader:488
      throw e;
      ^

Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './lib/cjs/flatgeobuf.js' is not defined by "exports" in /mnt/d/workspace/flatgeobuf_test/node_modules/flatgeobuf/package.json
    at new NodeError (node:internal/errors:371:5)
    at throwExportsNotFound (node:internal/modules/esm/resolve:440:9)
    at packageExportsResolve (node:internal/modules/esm/resolve:692:3)
    at resolveExports (node:internal/modules/cjs/loader:482:36)
    at Function.Module._findPath (node:internal/modules/cjs/loader:522:31)
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:919:27)
    at Function.Module._load (node:internal/modules/cjs/loader:778:27)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/mnt/d/workspace/flatgeobuf_test/mjs/server.cjs:5:17) {
  code: 'ERR_PACKAGE_PATH_NOT_EXPORTED'
}

Environment: Node: v16.13.0 npm: 8.1.4 os: Ubuntu-20.04 "flatgeobuf": "^3.20.1", "node-fetch": "^2.6.6"

const resource = 'https://gvpub.oss-cn-beijing.aliyuncs.com/UScounties.fgb';
const http = require('http');
const url = require('url');
const fetch  = require('node-fetch');
const geojson = require('flatgeobuf/lib/cjs/flatgeobuf.js');
const port = 3000;
let server = http.createServer(async (req, res) => {
    const urlObj = url.parse(req.url || '', true);
    const { pathname, query } = urlObj;
    if (pathname && pathname.startsWith('/api') && pathname === '/api/data') {
        if (query.bounds) {
        }
        else {
            return await getData();
        }
    }
    else
        res.write('test');
    res.end();
});
server.listen(port, () => {
    console.log(`Server is running on port: ${port}`);
});
async function getData() {
    const response = await fetch(resource, {});
    const features = [];
    console.log(response.body);
    const aa = geojson.deserialize(response.body);
    return features;
}

When I downgrade flatgeobuf to 3.17.0, and import it as below: const geojson = require("flatgeobuf/lib/cjs/geojson") it throws error too:

/mnt/d/workspace/aa_del/node_modules/flatgeobuf/lib/cjs/geojson.js:13
    else if (input instanceof ReadableStream)
                              ^

ReferenceError: ReadableStream is not defined
    at Object.deserialize (/mnt/d/workspace/aa_del/node_modules/flatgeobuf/lib/cjs/geojson.js:13:31)
    at getData (/mnt/d/workspace/aa_del/server.js:29:24)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Server.<anonymous> (/mnt/d/workspace/aa_del/server.js:15:20)

Can anyone give me some hint? Thanks! @bjornharrtell

Is there a way to log or access aggregate network usage for HttpFgbReader?

Specifically, I'd like to do something like:

println!("downloaded this many bytes: {}", reader.downloaded_byte_count())

background

I've been messing around with flatgeobuffers+geozero over the last day or so. Pretty nice so far!

vs. the previous geosjson implementation, obviously parsing is much faster and filesize is much less. 🚀

Previously my application downloaded a specific pre-sliced geojson file clipped to the client's specific map area. What I want to try is, instead of having to prepare pre-sliced files for each of my maps, have just one huge network hosted FGB file, and have the application self-serve what it needs using FGB's bbox selection feature (via HTTPFgbReader).

But I need to measure exactly how much network transfer is actually entailed in this (downloading the headers, the index, and just the feature slices it needs.)

The best I've found so far is to turn on debug logging and doing math with lines like:

[2020-12-16T17:06:07Z DEBUG hyper::proto::h1::conn] incoming body completed 
[2020-12-16T17:06:07Z DEBUG hyper::client::pool] pooling idle connection for ("https", s3.amazonaws.com) 
[2020-12-16T17:06:07Z DEBUG hyper::client::pool] reuse idle connection for ("https", s3.amazonaws.com) 
[2020-12-16T17:06:07Z DEBUG hyper::proto::h1::io] flushed 119 bytes 
[2020-12-16T17:06:07Z DEBUG hyper::proto::h1::io] read 431 bytes 
[2020-12-16T17:06:07Z DEBUG hyper::proto::h1::io] parsed 10 headers 
[2020-12-16T17:06:07Z DEBUG hyper::proto::h1::conn] incoming body is content-length (1048576 bytes) 
[2020-12-16T17:06:07Z DEBUG reqwest::async_impl::client] response '206 Partial Content' for https://s3.amazonaws.com/foo/bar_areas.fgb
[2020-12-16T17:06:07Z DEBUG hyper::proto::h1::io] read 8569 bytes 
...
[2020-12-16T17:06:07Z DEBUG hyper::proto::h1::conn] incoming body is content-length (1048576 bytes) 
[2020-12-16T17:06:07Z DEBUG reqwest::async_impl::client] response '206 Partial Content' for https://s3.amazonaws.com/foo/bar_areas.fgb
[2020-12-16T17:06:07Z DEBUG hyper::proto::h1::io] read 8569 bytes 
[2020-12-16T17:06:07Z DEBUG hyper::proto::h1::io] read 9000 bytes 
[2020-12-16T17:06:07Z DEBUG hyper::proto::h1::io] read 9000 bytes 
[2020-12-16T17:06:07Z DEBUG hyper::proto::h1::io] read 9000 bytes 
[2020-12-16T17:06:07Z DEBUG hyper::proto::h1::io] read 16384 bytes 
[2020-12-16T17:06:07Z DEBUG hyper::proto::h1::io] read 1024 bytes 
[2020-12-16T17:06:07Z DEBUG hyper::proto::h1::io] read 16384 bytes 
[2020-12-16T17:06:07Z DEBUG hyper::proto::h1::io] read 1024 bytes 
[2020-12-16T17:06:07Z DEBUG hyper::proto::h1::io] read 16384 bytes 
[2020-12-16T17:06:07Z DEBUG hyper::proto::h1::io] read 1024 bytes 
[2020-12-16T17:06:07Z DEBUG hyper::proto::h1::io] read 16384 bytes 
[2020-12-16T17:06:07Z DEBUG hyper::proto::h1::io] read 1024 bytes 

... is there a better way to get this in aggregate?

As per current spec the metadata for a column consists of only name and type (https://github.com/bjornharrtell/flatgeobuf/blob/bb80f92a6d8d6ff277da7b7d590529e8f0c1fe83/src/fbs/header.fbs#L44-L46). It could be helpful to optionally be able to provide more details, fx. length/width/precision and nullability.

This could probably be added without breaking spec v3 backward compatibility.

Very large text data are stored as they are, without compression:

Example: sorry in Japanese

Is it possible to store compressed (by zip or so?) data if text size were larger than threshold? Is this silly idea?

Hi

Sorry if this topic has been discussed a lot in the past.

I found Object or array type values are serialized not as binary but as JSON text.

Example: Sorry in Japanese

I think this greatly hinders the improvement of the compression ratio. Can I expect the function to binary serialize attribute values ​​of Object and Array types in future?

Hey 👋

We're currently trying to integrate a self-hosted, indexed FlatGeobuf in our site (exported from QGIS). For the site, we're using the filtering feature as demonstrated in the https://flatgeobuf.org/examples/leaflet/large.html example.

Here's our minimum working example: https://atlas.thuenen.de/webspace/agraratlas/rs/test_fgb_vti.html

While everything works as expected in Chrome, Safari and Edge, it seems that Firefox attempts to download the entire .fgb file instead of just the parts contained in the bounding box. Since the example provided in leaflet/large.html also works in Firefox on our end, it seems that there's something wrong with our file/server, but we couldn't figure it out so far.

Any ideas of what we're doing wrong? Thanks for your help!

Was hoping to be able to place flatgeobuf into a webworker and provide range requests/filtered requests from arbitrary serialized geojson features.

However, the range request feature with spatial filter seems directly tied to the HTTP/request system.

Reviewing the code, it looks tightly coupled.

It seems like it should just be given an interface that returns various slices of data from . Is it possible to make a cheap object interface that can be substituted for the <url> input.

I stumbled on this -- if you make a simple app using flatgeobuf that doesn't depend directly on [email protected], it will fail to handle https urls with this error message:

Error: http error `error sending request for url (https://blah/blah/blah.fgb): error trying to connect: invalid URL, scheme is not http`

That error originates in hyper. I think the issue is that http-range-client has an optional dependency on reqwest and so without that, you'll get a client that has no support for HTTPS. That's...really odd. I'd suggest either:

• document clearly that HttpFgbReader will not read HTTPS urls unless you add an explicit dependency on the same version of request that flatgeobuf's http-range-client depends on, or
• add a tls or http feature to flatgeobuf and document that in the README.