sections
478 rows sorted by content
This data as json, CSV (advanced)
id | page | ref | title | content ▼ | breadcrumbs | references |
---|---|---|---|---|---|---|
publish:cli-package | publish | cli-package | datasette package | If you have docker installed (e.g. using Docker for Mac ) you can use the datasette package command to create a new Docker image in your local repository containing the datasette app bundled together with one or more SQLite databases: datasette package mydatabase.db Here's example output for the package command: $ datasette package parlgov.db --extra-options="--setting sql_time_limit_ms 2500" Sending build context to Docker daemon 4.459MB Step 1/7 : FROM python:3.11.0-slim-bullseye ---> 79e1dc9af1c1 Step 2/7 : COPY . /app ---> Using cache ---> cd4ec67de656 Step 3/7 : WORKDIR /app ---> Using cache ---> 139699e91621 Step 4/7 : RUN pip install datasette ---> Using cache ---> 340efa82bfd7 Step 5/7 : RUN datasette inspect parlgov.db --inspect-file inspect-data.json ---> Using cache ---> 5fddbe990314 Step 6/7 : EXPOSE 8001 ---> Using cache ---> 8e83844b0fed Step 7/7 : CMD datasette serve parlgov.db --port 8001 --inspect-file inspect-data.json --setting sql_time_limit_ms 2500 ---> Using cache ---> 1bd380ea8af3 Successfully built 1bd380ea8af3 You can now run the resulting container like so: docker run -p 8081:8001 1bd380ea8af3 This exposes port 8001 inside the container as port 8081 on your host machine, so you can access the application at http://localhost:8081/ You can customize the port that is exposed by the container using the --port option: datasette package mydatabase.db --port 8080 A full list of options can be seen by running datasette package --help : See datasette package for the full list of options for this command. | ["Publishing data"] | [{"href": "https://www.docker.com/docker-mac", "label": "Docker for Mac"}] |
installation:id1 | installation | id1 | Installation | If you just want to try Datasette out you don't need to install anything: see Try Datasette without installing anything using Glitch There are two main options for installing Datasette. You can install it directly on to your machine, or you can install it using Docker. If you want to start making contributions to the Datasette project by installing a copy that lets you directly modify the code, take a look at our guide to Setting up a development environment . Basic installation Datasette Desktop for Mac Using Homebrew Using pip Advanced installation options Using pipx Installing plugins using pipx Upgrading packages using pipx Using Docker Loading SpatiaLite Installing plugins A note about extensions | [] | [] |
performance:performance-hashed-urls | performance | performance-hashed-urls | datasette-hashed-urls | If you open a database file in immutable mode using the -i option, you can be assured that the content of that database will not change for the lifetime of the Datasette server. The datasette-hashed-urls plugin implements an optimization where your database is served with part of the SHA-256 hash of the database contents baked into the URL. A database at /fixtures will instead be served at /fixtures-aa7318b , and a year-long cache expiry header will be returned with those pages. This will then be cached by both browsers and caching proxies such as Cloudflare or Fastly, providing a potentially significant performance boost. To install the plugin, run the following: datasette install datasette-hashed-urls Prior to Datasette 0.61 hashed URL mode was a core Datasette feature, enabled using the hash_urls setting. This implementation has now been removed in favor of the datasette-hashed-urls plugin. Prior to Datasette 0.28 hashed URL mode was the default behaviour for Datasette, since all database files were assumed to be immutable and unchanging. From 0.28 onwards the default has been to treat database files as mutable unless explicitly configured otherwise. | ["Performance and caching"] | [{"href": "https://datasette.io/plugins/datasette-hashed-urls", "label": "datasette-hashed-urls plugin"}] |
sql_queries:sql-views | sql_queries | sql-views | Views | If you want to bundle some pre-written SQL queries with your Datasette-hosted database you can do so in two ways. The first is to include SQL views in your database - Datasette will then list those views on your database index page. The quickest way to create views is with the SQLite command-line interface: $ sqlite3 sf-trees.db SQLite version 3.19.3 2017-06-27 16:48:08 Enter ".help" for usage hints. sqlite> CREATE VIEW demo_view AS select qSpecies from Street_Tree_List; <CTRL+D> | ["Running SQL queries"] | [] |
changelog:v0-29-medium-changes | changelog | v0-29-medium-changes | Easier custom templates for table rows | If you want to customize the display of individual table rows, you can do so using a _table.html template include that looks something like this: {% for row in display_rows %} <div> <h2>{{ row["title"] }}</h2> <p>{{ row["description"] }}<lp> <p>Category: {{ row.display("category_id") }}</p> </div> {% endfor %} This is a backwards incompatible change . If you previously had a custom template called _rows_and_columns.html you need to rename it to _table.html . See Custom templates for full details. | ["Changelog", "0.29 (2019-07-07)"] | [] |
installation:installing-plugins | installation | installing-plugins | Installing plugins | If you want to install plugins into your local Datasette Docker image you can do so using the following recipe. This will install the plugins and then save a brand new local image called datasette-with-plugins : docker run datasetteproject/datasette \ pip install datasette-vega docker commit $(docker ps -lq) datasette-with-plugins You can now run the new custom image like so: docker run -p 8001:8001 -v `pwd`:/mnt \ datasette-with-plugins \ datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db You can confirm that the plugins are installed by visiting http://127.0.0.1:8001/-/plugins Some plugins such as datasette-ripgrep may need additional system packages. You can install these by running apt-get install inside the container: docker run datasette-057a0 bash -c ' apt-get update && apt-get install ripgrep && pip install datasette-ripgrep' docker commit $(docker ps -lq) datasette-with-ripgrep | ["Installation", "Advanced installation options", "Using Docker"] | [{"href": "http://127.0.0.1:8001/-/plugins", "label": "http://127.0.0.1:8001/-/plugins"}, {"href": "https://datasette.io/plugins/datasette-ripgrep", "label": "datasette-ripgrep"}] |
facets:id2 | facets | id2 | Facet by JSON array | If your SQLite installation provides the json1 extension (you can check using /-/versions ) Datasette will automatically detect columns that contain JSON arrays of values and offer a faceting interface against those columns. This is useful for modelling things like tags without needing to break them out into a new table. Example here: latest.datasette.io/fixtures/facetable?_facet_array=tags | ["Facets"] | [{"href": "https://latest.datasette.io/fixtures/facetable?_facet_array=tags", "label": "latest.datasette.io/fixtures/facetable?_facet_array=tags"}] |
internals:internals-tracer-trace-child-tasks | internals | internals-tracer-trace-child-tasks | Tracing child tasks | If your code uses a mechanism such as asyncio.gather() to execute code in additional tasks you may find that some of the traces are missing from the display. You can use the trace_child_tasks() context manager to ensure these child tasks are correctly handled. from datasette import tracer with tracer.trace_child_tasks(): results = await asyncio.gather( # ... async tasks here ) This example uses the register_routes() plugin hook to add a page at /parallel-queries which executes two SQL queries in parallel using asyncio.gather() and returns their results. from datasette import hookimpl from datasette import tracer @hookimpl def register_routes(): async def parallel_queries(datasette): db = datasette.get_database() with tracer.trace_child_tasks(): one, two = await asyncio.gather( db.execute("select 1"), db.execute("select 2"), ) return Response.json( { "one": one.single_value(), "two": two.single_value(), } ) return [ (r"/parallel-queries$", parallel_queries), ] Adding ?_trace=1 will show that the trace covers both of those child tasks. | ["Internals for plugins", "datasette.tracer"] | [] |
full_text_search:configuring-fts-using-csvs-to-sqlite | full_text_search | configuring-fts-using-csvs-to-sqlite | Configuring FTS using csvs-to-sqlite | If your data starts out in CSV files, you can use Datasette's companion tool csvs-to-sqlite to convert that file into a SQLite database and enable full-text search on specific columns. For a file called items.csv where you want full-text search to operate against the name and description columns you would run the following: $ csvs-to-sqlite items.csv items.db -f name -f description | ["Full-text search", "Enabling full-text search for a SQLite table"] | [{"href": "https://github.com/simonw/csvs-to-sqlite", "label": "csvs-to-sqlite"}] |
performance:http-caching | performance | http-caching | HTTP caching | If your database is immutable and guaranteed not to change, you can gain major performance improvements from Datasette by enabling HTTP caching. This can work at two different levels. First, it can tell browsers to cache the results of queries and serve future requests from the browser cache. More significantly, it allows you to run Datasette behind a caching proxy such as Varnish or use a cache provided by a hosted service such as Fastly or Cloudflare . This can provide incredible speed-ups since a query only needs to be executed by Datasette the first time it is accessed - all subsequent hits can then be served by the cache. Using a caching proxy in this way could enable a Datasette-backed visualization to serve thousands of hits a second while running Datasette itself on extremely inexpensive hosting. Datasette's integration with HTTP caches can be enabled using a combination of configuration options and query string arguments. The default_cache_ttl setting sets the default HTTP cache TTL for all Datasette pages. This is 5 seconds unless you change it - you can set it to 0 if you wish to disable HTTP caching entirely. You can also change the cache timeout on a per-request basis using the ?_ttl=10 query string parameter. This can be useful when you are working with the Datasette JSON API - you may decide that a specific query can be cached for a longer time, or maybe you need to set ?_ttl=0 for some requests for example if you are running a SQL order by random() query. | ["Performance and caching"] | [{"href": "https://varnish-cache.org/", "label": "Varnish"}, {"href": "https://www.fastly.com/", "label": "Fastly"}, {"href": "https://www.cloudflare.com/", "label": "Cloudflare"}] |
writing_plugins:writing-plugins-static-assets | writing_plugins | writing-plugins-static-assets | Static assets | If your plugin has a static/ directory, Datasette will automatically configure itself to serve those static assets from the following path: /-/static-plugins/NAME_OF_PLUGIN_PACKAGE/yourfile.js Use the datasette.urls.static_plugins(plugin_name, path) method to generate URLs to that asset that take the base_url setting into account, see datasette.urls . To bundle the static assets for a plugin in the package that you publish to PyPI, add the following to the plugin's setup.py : package_data = ( { "datasette_plugin_name": [ "static/plugin.js", ], }, ) Where datasette_plugin_name is the name of the plugin package (note that it uses underscores, not hyphens) and static/plugin.js is the path within that package to the static file. datasette-cluster-map is a useful example of a plugin that includes packaged static assets in this way. | ["Writing plugins"] | [{"href": "https://github.com/simonw/datasette-cluster-map", "label": "datasette-cluster-map"}] |
writing_plugins:writing-plugins-custom-templates | writing_plugins | writing-plugins-custom-templates | Custom templates | If your plugin has a templates/ directory, Datasette will attempt to load templates from that directory before it uses its own default templates. The priority order for template loading is: templates from the --template-dir argument, if specified templates from the templates/ directory in any installed plugins default templates that ship with Datasette See Custom pages and templates for more details on how to write custom templates, including which filenames to use to customize which parts of the Datasette UI. Templates should be bundled for distribution using the same package_data mechanism in setup.py described for static assets above, for example: package_data = ( { "datasette_plugin_name": [ "templates/my_template.html", ], }, ) You can also use wildcards here such as templates/*.html . See datasette-edit-schema for an example of this pattern. | ["Writing plugins"] | [{"href": "https://github.com/simonw/datasette-edit-schema", "label": "datasette-edit-schema"}] |
testing_plugins:testing-plugins-pytest-httpx | testing_plugins | testing-plugins-pytest-httpx | Testing outbound HTTP calls with pytest-httpx | If your plugin makes outbound HTTP calls - for example datasette-auth-github or datasette-import-table - you may need to mock those HTTP requests in your tests. The pytest-httpx package is a useful library for mocking calls. It can be tricky to use with Datasette though since it mocks all HTTPX requests, and Datasette's own testing mechanism uses HTTPX internally. To avoid breaking your tests, you can return ["localhost"] from the non_mocked_hosts() fixture. As an example, here's a very simple plugin which executes an HTTP response and returns the resulting content: from datasette import hookimpl from datasette.utils.asgi import Response import httpx @hookimpl def register_routes(): return [ (r"^/-/fetch-url$", fetch_url), ] async def fetch_url(datasette, request): if request.method == "GET": return Response.html( """ <form action="/-/fetch-url" method="post"> <input type="hidden" name="csrftoken" value="{}"> <input name="url"><input type="submit"> </form>""".format( request.scope["csrftoken"]() ) ) vars = await request.post_vars() url = vars["url"] return Response.text(httpx.get(url).text) Here's a test for that plugin that mocks the HTTPX outbound request: from datasette.app import Datasette import pytest @pytest.fixture def non_mocked_hosts(): # This ensures httpx-mock will not affect Datasette's own # httpx calls made in the tests by datasette.client: return ["localhost"] async def test_outbound_http_call(httpx_mock): httpx_mock.add_response( url="https://www.example.com/", text="Hello world", ) datasette = Datasette([], memory=True) response = await datasette.client.post( "/-/fetch-url", data={"url": "https://www.example.com/"}, ) assert response.text == "Hello world" outbound_request = httpx_mock.get_request()… | ["Testing plugins"] | [{"href": "https://pypi.org/project/pytest-httpx/", "label": "pytest-httpx"}] |
changelog:id42 | changelog | id42 | 0.51.1 (2020-10-31) | Improvements to the new Binary data documentation page. | ["Changelog"] | [] |
internals:internals-response-asgi-send | internals | internals-response-asgi-send | Returning a response with .asgi_send(send) | In most cases you will return Response objects from your own view functions. You can also use a Response instance to respond at a lower level via ASGI, for example if you are writing code that uses the asgi_wrapper(datasette) hook. Create a Response object and then use await response.asgi_send(send) , passing the ASGI send function. For example: async def require_authorization(scope, receive, send): response = Response.text( "401 Authorization Required", headers={ "www-authenticate": 'Basic realm="Datasette", charset="UTF-8"' }, status=401, ) await response.asgi_send(send) | ["Internals for plugins", "Response class"] | [] |
changelog:id17 | changelog | id17 | 0.61 (2022-03-23) | In preparation for Datasette 1.0, this release includes two potentially backwards-incompatible changes. Hashed URL mode has been moved to a separate plugin, and the way Datasette generates URLs to databases and tables with special characters in their name such as / and . has changed. Datasette also now requires Python 3.7 or higher. URLs within Datasette now use a different encoding scheme for tables or databases that include "special" characters outside of the range of a-zA-Z0-9_- . This scheme is explained here: Tilde encoding . ( #1657 ) Removed hashed URL mode from Datasette. The new datasette-hashed-urls plugin can be used to achieve the same result, see datasette-hashed-urls for details. ( #1661 ) Databases can now have a custom path within the Datasette instance that is independent of the database name, using the db.route property. ( #1668 ) Datasette is now covered by a Code of Conduct . ( #1654 ) Python 3.6 is no longer supported. ( #1577 ) Tests now run against Python 3.11-dev. ( #1621 ) New datasette.ensure_permissions(actor, permissions) internal method for checking multiple permissions at once. ( #1675 ) New datasette.check_visibility(actor, action, resource=None) internal method for checking if a user can see a resource that would otherwise be invisible to unauthenticated users. ( #1678 ) Table and row HTML pages now include a <link rel="alternate" type="application/json+datasette" href="..."> element and return a Link: URL; rel="alternate"; type="applicatio… | ["Changelog"] | [{"href": "https://github.com/simonw/datasette/issues/1657", "label": "#1657"}, {"href": "https://github.com/simonw/datasette/issues/1661", "label": "#1661"}, {"href": "https://github.com/simonw/datasette/issues/1668", "label": "#1668"}, {"href": "https://github.com/simonw/datasette/blob/main/CODE_OF_CONDUCT.md", "label": "Code of Conduct"}, {"href": "https://github.com/simonw/datasette/issues/1654", "label": "#1654"}, {"href": "https://github.com/simonw/datasette/issues/1577", "label": "#1577"}, {"href": "https://github.com/simonw/datasette/issues/1621", "label": "#1621"}, {"href": "https://github.com/simonw/datasette/issues/1675", "label": "#1675"}, {"href": "https://github.com/simonw/datasette/issues/1678", "label": "#1678"}, {"href": "https://github.com/simonw/datasette/issues/1533", "label": "#1533"}, {"href": "https://github.com/simonw/datasette/issues/1612", "label": "#1612"}, {"href": "https://github.com/simonw/datasette/issues/1603", "label": "#1603"}, {"href": "https://github.com/simonw/datasette/issues/1587", "label": "#1587"}, {"href": "https://github.com/simonw/datasette/issues/1601", "label": "#1601"}, {"href": "https://github.com/simonw/datasette/issues/1576", "label": "#1576"}, {"href": "https://github.com/simonw/datasette/issues/957", "label": "#957"}, {"href": "https://github.com/simonw/datasette/issues/1607", "label": "#1607"}, {"href": "https://datasette.io/tutorials", "label": "Datasette Tutorials"}, {"href": "https://github.com/simonw/datasette/pull/1649", "label": "#1649"}, {"href": "https://github.com/simonw/datasette/issues/1545", "label": "#1545"}, {"href": "https://github.com/simonw/datasette/issues/1228", "label": "#1228"}] |
settings:setting-truncate-cells-html | settings | setting-truncate-cells-html | truncate_cells_html | In the HTML table view, truncate any strings that are longer than this value. The full value will still be available in CSV, JSON and on the individual row HTML page. Set this to 0 to disable truncation. datasette mydatabase.db --setting truncate_cells_html 0 | ["Settings", "Settings"] | [] |
cli-reference:cli-help-install-help | cli-reference | cli-help-install-help | datasette install | Install new Datasette plugins. This command works like pip install but ensures that your plugins will be installed into the same environment as Datasette. This command: datasette install datasette-cluster-map Would install the datasette-cluster-map plugin. [[[cog help(["install", "--help"]) ]]] Usage: datasette install [OPTIONS] PACKAGES... Install plugins and packages from PyPI into the same environment as Datasette Options: -U, --upgrade Upgrade packages to latest version --help Show this message and exit. [[[end]]] | ["CLI reference"] | [{"href": "https://datasette.io/plugins/datasette-cluster-map", "label": "datasette-cluster-map"}] |
internals:internals-database | internals | internals-database | Database class | Instances of the Database class can be used to execute queries against attached SQLite databases, and to run introspection against their schemas. | ["Internals for plugins"] | [] |
changelog:javascript-modules | changelog | javascript-modules | JavaScript modules | JavaScript modules were introduced in ECMAScript 2015 and provide native browser support for the import and export keywords. To use modules, JavaScript needs to be included in <script> tags with a type="module" attribute. Datasette now has the ability to output <script type="module"> in places where you may wish to take advantage of modules. The extra_js_urls option described in Custom CSS and JavaScript can now be used with modules, and module support is also available for the extra_body_script() plugin hook. ( #1186 , #1187 ) datasette-leaflet-freedraw is the first example of a Datasette plugin that takes advantage of the new support for JavaScript modules. See Drawing shapes on a map to query a SpatiaLite database for more on this plugin. | ["Changelog", "0.54 (2021-01-25)"] | [{"href": "https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules", "label": "JavaScript modules"}, {"href": "https://github.com/simonw/datasette/issues/1186", "label": "#1186"}, {"href": "https://github.com/simonw/datasette/issues/1187", "label": "#1187"}, {"href": "https://datasette.io/plugins/datasette-leaflet-freedraw", "label": "datasette-leaflet-freedraw"}, {"href": "https://simonwillison.net/2021/Jan/24/drawing-shapes-spatialite/", "label": "Drawing shapes on a map to query a SpatiaLite database"}] |
plugin_hooks:plugin-hook-render-cell | plugin_hooks | plugin-hook-render-cell | render_cell(row, value, column, table, database, datasette) | Lets you customize the display of values within table cells in the HTML table view. row - sqlite.Row The SQLite row object that the value being rendered is part of value - string, integer, float, bytes or None The value that was loaded from the database column - string The name of the column being rendered table - string or None The name of the table - or None if this is a custom SQL query database - string The name of the database datasette - Datasette class You can use this to access plugin configuration options via datasette.plugin_config(your_plugin_name) , or to execute SQL queries. If your hook returns None , it will be ignored. Use this to indicate that your hook is not able to custom render this particular value. If the hook returns a string, that string will be rendered in the table cell. If you want to return HTML markup you can do so by returning a jinja2.Markup object. You can also return an awaitable function which returns a value. Datasette will loop through all available render_cell hooks and display the value returned by the first one that does not return None . Here is an example of a custom render_cell() plugin wh… | ["Plugin hooks"] | [{"href": "https://datasette.io/plugins/datasette-render-binary", "label": "datasette-render-binary"}, {"href": "https://datasette.io/plugins/datasette-render-markdown", "label": "datasette-render-markdown"}, {"href": "https://datasette.io/plugins/datasette-json-html", "label": "datasette-json-html"}] |
internals:database-execute-write-script | internals | database-execute-write-script | await db.execute_write_script(sql, block=True) | Like execute_write() but can be used to send multiple SQL statements in a single string separated by semicolons, using the sqlite3 conn.executescript() method. | ["Internals for plugins", "Database class"] | [{"href": "https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor.executescript", "label": "conn.executescript()"}] |
internals:database-execute-write-many | internals | database-execute-write-many | await db.execute_write_many(sql, params_seq, block=True) | Like execute_write() but uses the sqlite3 conn.executemany() method. This will efficiently execute the same SQL statement against each of the parameters in the params_seq iterator, for example: await db.execute_write_many( "insert into characters (id, name) values (?, ?)", [(1, "Melanie"), (2, "Selma"), (2, "Viktor")], ) | ["Internals for plugins", "Database class"] | [{"href": "https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor.executemany", "label": "conn.executemany()"}] |
changelog:other-small-fixes | changelog | other-small-fixes | Other small fixes | Made several performance improvements to the database schema introspection code that runs when Datasette first starts up. ( #1555 ) Label columns detected for foreign keys are now case-insensitive, so Name or TITLE will be detected in the same way as name or title . ( #1544 ) Upgraded Pluggy dependency to 1.0. ( #1575 ) Now using Plausible analytics for the Datasette documentation. explain query plan is now allowed with varying amounts of whitespace in the query. ( #1588 ) New CLI reference page showing the output of --help for each of the datasette sub-commands. This lead to several small improvements to the help copy. ( #1594 ) Fixed bug where writable canned queries could not be used with custom templates. ( #1547 ) Improved fix for a bug where columns with a underscore prefix could result in unnecessary hidden form fields. ( #1527 ) | ["Changelog", "0.60 (2022-01-13)"] | [{"href": "https://github.com/simonw/datasette/issues/1555", "label": "#1555"}, {"href": "https://github.com/simonw/datasette/issues/1544", "label": "#1544"}, {"href": "https://github.com/simonw/datasette/issues/1575", "label": "#1575"}, {"href": "https://plausible.io/", "label": "Plausible analytics"}, {"href": "https://github.com/simonw/datasette/issues/1588", "label": "#1588"}, {"href": "https://github.com/simonw/datasette/issues/1594", "label": "#1594"}, {"href": "https://github.com/simonw/datasette/issues/1547", "label": "#1547"}, {"href": "https://github.com/simonw/datasette/issues/1527", "label": "#1527"}] |
internals:internals | internals | internals | Internals for plugins | Many Plugin hooks are passed objects that provide access to internal Datasette functionality. The interface to these objects should not be considered stable with the exception of methods that are documented here. | [] | [] |
settings:setting-num-sql-threads | settings | setting-num-sql-threads | num_sql_threads | Maximum number of threads in the thread pool Datasette uses to execute SQLite queries. Defaults to 3. datasette mydatabase.db --setting num_sql_threads 10 Setting this to 0 turns off threaded SQL queries entirely - useful for environments that do not support threading such as Pyodide . | ["Settings", "Settings"] | [{"href": "https://pyodide.org/", "label": "Pyodide"}] |
metadata:per-database-and-per-table-metadata | metadata | per-database-and-per-table-metadata | Per-database and per-table metadata | Metadata at the top level of the JSON will be shown on the index page and in the footer on every page of the site. The license and source is expected to apply to all of your data. You can also provide metadata at the per-database or per-table level, like this: { "databases": { "database1": { "source": "Alternative source", "source_url": "http://example.com/", "tables": { "example_table": { "description_html": "Custom <em>table</em> description", "license": "CC BY 3.0 US", "license_url": "https://creativecommons.org/licenses/by/3.0/us/" } } } } } Each of the top-level metadata fields can be used at the database and table level. | ["Metadata"] | [] |
changelog:id100 | changelog | id100 | 0.23.2 (2018-07-07) | Minor bugfix and documentation release. CSV export now respects --cors , fixes #326 Installation instructions , including docker image - closes #328 Fix for row pages for tables with / in, closes #325 | ["Changelog"] | [{"href": "https://github.com/simonw/datasette/issues/326", "label": "#326"}, {"href": "https://github.com/simonw/datasette/issues/328", "label": "#328"}, {"href": "https://github.com/simonw/datasette/issues/325", "label": "#325"}] |
changelog:id104 | changelog | id104 | 0.23.1 (2018-06-21) | Minor bugfix release. Correctly display empty strings in HTML table, closes #314 Allow "." in database filenames, closes #302 404s ending in slash redirect to remove that slash, closes #309 Fixed incorrect display of compound primary keys with foreign key references. Closes #319 Docs + example of canned SQL query using || concatenation. Closes #321 Correctly display facets with value of 0 - closes #318 Default 'expand labels' to checked in CSV advanced export | ["Changelog"] | [{"href": "https://github.com/simonw/datasette/issues/314", "label": "#314"}, {"href": "https://github.com/simonw/datasette/issues/302", "label": "#302"}, {"href": "https://github.com/simonw/datasette/issues/309", "label": "#309"}, {"href": "https://github.com/simonw/datasette/issues/319", "label": "#319"}, {"href": "https://github.com/simonw/datasette/issues/321", "label": "#321"}, {"href": "https://github.com/simonw/datasette/issues/318", "label": "#318"}] |
json_api:json-api-discover-alternate | json_api | json-api-discover-alternate | Discovering the JSON for a page | Most of the HTML pages served by Datasette provide a mechanism for discovering their JSON equivalents using the HTML link mechanism. You can find this near the top of the source code of those pages, looking like this: <link rel="alternate" type="application/json+datasette" href="https://latest.datasette.io/fixtures/sortable.json"> The JSON URL is also made available in a Link HTTP header for the page: Link: https://latest.datasette.io/fixtures/sortable.json; rel="alternate"; type="application/json+datasette" | ["JSON API"] | [] |
changelog:id133 | changelog | id133 | 0.20 (2018-04-20) | Mostly new work on the Plugins mechanism: plugins can now bundle static assets and custom templates, and datasette publish has a new --install=name-of-plugin option. Add col-X classes to HTML table on custom query page Fixed out-dated template in documentation Plugins can now bundle custom templates, #224 Added /-/metadata /-/plugins /-/inspect, #225 Documentation for --install option, refs #223 Datasette publish/package --install option, #223 Fix for plugins in Python 3.5, #222 New plugin hooks: extra_css_urls() and extra_js_urls(), #214 /-/static-plugins/PLUGIN_NAME/ now serves static/ from plugins <th> now gets class="col-X" - plus added col-X documentation Use to_css_class for table cell column classes This ensures that columns with spaces in the name will still generate usable CSS class names. Refs #209 Add column name classes to <td>s, make PK bold [Russ Garrett] Don't duplicate simple primary keys in the link column [Russ Garrett] When there's a simple (single-column) primary key, it looks weird to duplicate it in the link column. This change removes the second PK column and treats the link column as if it were the PK colu… | ["Changelog"] | [{"href": "https://github.com/simonw/datasette/issues/224", "label": "#224"}, {"href": "https://github.com/simonw/datasette/issues/225", "label": "#225"}, {"href": "https://github.com/simonw/datasette/issues/223", "label": "#223"}, {"href": "https://github.com/simonw/datasette/issues/223", "label": "#223"}, {"href": "https://github.com/simonw/datasette/issues/222", "label": "#222"}, {"href": "https://github.com/simonw/datasette/issues/214", "label": "#214"}, {"href": "https://github.com/simonw/datasette/issues/209", "label": "#209"}, {"href": "https://github.com/simonw/datasette/issues/209", "label": "#209"}] |
sql_queries:canned-queries-magic-parameters | sql_queries | canned-queries-magic-parameters | Magic parameters | Named parameters that start with an underscore are special: they can be used to automatically add values created by Datasette that are not contained in the incoming form fields or query string. These magic parameters are only supported for canned queries: to avoid security issues (such as queries that extract the user's private cookies) they are not available to SQL that is executed by the user as a custom SQL query. Available magic parameters are: _actor_* - e.g. _actor_id , _actor_name Fields from the currently authenticated Actors . _header_* - e.g. _header_user_agent Header from the incoming HTTP request. The key should be in lower case and with hyphens converted to underscores e.g. _header_user_agent or _header_accept_language . _cookie_* - e.g. _cookie_lang The value of the incoming cookie of that name. _now_epoch The number of seconds since the Unix epoch. _now_date_utc The date in UTC, e.g. 2020-06-01 _now_datetime_utc The ISO 8601 datetime in UTC, e.g. 2020-06-24T18:01:07Z _random_chars_* - e.g. … | ["Running SQL queries", "Canned queries"] | [] |
changelog:id63 | changelog | id63 | 0.39 (2020-03-24) | New base_url configuration setting for serving up the correct links while running Datasette under a different URL prefix. ( #394 ) New metadata settings "sort" and "sort_desc" for setting the default sort order for a table. See Setting a default sort order . ( #702 ) Sort direction arrow now displays by default on the primary key. This means you only have to click once (not twice) to sort in reverse order. ( #677 ) New await Request(scope, receive).post_vars() method for accessing POST form variables. ( #700 ) Plugin hooks documentation now links to example uses of each plugin. ( #709 ) | ["Changelog"] | [{"href": "https://github.com/simonw/datasette/issues/394", "label": "#394"}, {"href": "https://github.com/simonw/datasette/issues/702", "label": "#702"}, {"href": "https://github.com/simonw/datasette/issues/677", "label": "#677"}, {"href": "https://github.com/simonw/datasette/issues/700", "label": "#700"}, {"href": "https://github.com/simonw/datasette/issues/709", "label": "#709"}] |
changelog:id27 | changelog | id27 | 0.58 (2021-07-14) | New datasette --uds /tmp/datasette.sock option for binding Datasette to a Unix domain socket, see proxy documentation ( #1388 ) "searchmode": "raw" table metadata option for defaulting a table to executing SQLite full-text search syntax without first escaping it, see Advanced SQLite search queries . ( #1389 ) New plugin hook: get_metadata(datasette, key, database, table) , for returning custom metadata for an instance, database or table. Thanks, Brandon Roberts! ( #1384 ) New plugin hook: skip_csrf(datasette, scope) , for opting out of CSRF protection based on the incoming request. ( #1377 ) The menu_links() , table_actions() and database_actions() plugin hooks all gained a new optional request argument providing access to the current request. ( #1371 ) Major performance improvement for Datasette faceting. ( #1394 ) Improved documentation for Running Datasette behind a proxy to recommend using ProxyPreservehost On with Apache. ( #1387 ) POST requests to endpoints that do not support that HTTP verb now return a 405 error. db.path can now be provided as a pathlib.Path object, useful when writing unit tests for plugins. Thanks, Chris Amico. ( #1365 ) | ["Changelog"] | [{"href": "https://github.com/simonw/datasette/issues/1388", "label": "#1388"}, {"href": "https://github.com/simonw/datasette/issues/1389", "label": "#1389"}, {"href": "https://github.com/simonw/datasette/issues/1384", "label": "#1384"}, {"href": "https://github.com/simonw/datasette/issues/1377", "label": "#1377"}, {"href": "https://github.com/simonw/datasette/issues/1371", "label": "#1371"}, {"href": "https://github.com/simonw/datasette/issues/1394", "label": "#1394"}, {"href": "https://github.com/simonw/datasette/issues/1387", "label": "#1387"}, {"href": "https://github.com/simonw/datasette/issues/1365", "label": "#1365"}] |
changelog:id120 | changelog | id120 | 0.21 (2018-05-05) | New JSON _shape= options, the ability to set table _size= and a mechanism for searching within specific columns. Default tests to using a longer timelimit Every now and then a test will fail in Travis CI on Python 3.5 because it hit the default 20ms SQL time limit. Test fixtures now default to a 200ms time limit, and we only use the 20ms time limit for the specific test that tests query interruption. This should make our tests on Python 3.5 in Travis much more stable. Support _search_COLUMN=text searches, closes #237 Show version on /-/plugins page, closes #248 ?_size=max option, closes #249 Added /-/versions and /-/versions.json , closes #244 Sample output: { "python": { "version": "3.6.3", "full": "3.6.3 (default, Oct 4 2017, 06:09:38) \n[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)]" }, "datasette": { "version": "0.20" }, "sqlite": { "version": "3.23.1", "extensions": { "json1": null, "spatialite": "4.3.0a" } } } Renamed ?_sql_time_limit_ms= to ?_timelimit , closes #242 New ?_shape=array option + tweaks to _shape , closes #245 Default is now ?_shape=arrays (renamed from lists ) New ?_shape=array returns an array of objects as the root object … | ["Changelog"] | [{"href": "https://github.com/simonw/datasette/issues/237", "label": "#237"}, {"href": "https://github.com/simonw/datasette/issues/248", "label": "#248"}, {"href": "https://github.com/simonw/datasette/issues/249", "label": "#249"}, {"href": "https://github.com/simonw/datasette/issues/244", "label": "#244"}, {"href": "https://github.com/simonw/datasette/issues/242", "label": "#242"}, {"href": "https://github.com/simonw/datasette/issues/245", "label": "#245"}, {"href": "https://github.com/simonw/datasette/issues/240", "label": "#240"}, {"href": "https://github.com/simonw/datasette/issues/229", "label": "#229"}, {"href": "https://github.com/simonw/datasette/issues/230", "label": "#230"}, {"href": "https://github.com/simonw/datasette/issues/239", "label": "#239"}, {"href": "https://github.com/simonw/datasette/issues/228", "label": "#228"}, {"href": "https://github.com/simonw/datasette/issues/234", "label": "#234"}] |
changelog:id85 | changelog | id85 | 0.27 (2019-01-31) | New command: datasette plugins ( documentation ) shows you the currently installed list of plugins. Datasette can now output newline-delimited JSON using the new ?_shape=array&_nl=on query string option. Added documentation on The Datasette Ecosystem . Now using Python 3.7.2 as the base for the official Datasette Docker image . | ["Changelog"] | [{"href": "http://ndjson.org/", "label": "newline-delimited JSON"}, {"href": "https://hub.docker.com/r/datasetteproject/datasette/", "label": "Datasette Docker image"}] |
changelog:id58 | changelog | id58 | Smaller changes | New internals documentation for Request object and Response class . ( #706 ) request.url now respects the force_https_urls config setting. closes ( #781 ) request.args.getlist() returns [] if missing. Removed request.raw_args entirely. ( #774 ) New datasette.get_database() method. Added _ prefix to many private, undocumented methods of the Datasette class. ( #576 ) Removed the db.get_outbound_foreign_keys() method which duplicated the behaviour of db.foreign_keys_for_table() . New await datasette.permission_allowed() method. /-/actor debugging endpoint for viewing the currently authenticated actor. New request.cookies property. /-/plugins endpoint now shows a list of hooks implemented by each plugin, e.g. https://latest.datasette.io/-/plugins?all=1 request.post_vars() method no longer discards empty values. New "params" canned query key for explicitly setting named parameters, see Canned query parameters . ( #797 ) request.args is now a MultiParams object. Fixed a bug with the datasette plugins command. ( #802 ) Nicer pa… | ["Changelog", "0.44 (2020-06-11)"] | [{"href": "https://github.com/simonw/datasette/issues/706", "label": "#706"}, {"href": "https://github.com/simonw/datasette/issues/781", "label": "#781"}, {"href": "https://github.com/simonw/datasette/issues/774", "label": "#774"}, {"href": "https://github.com/simonw/datasette/issues/576", "label": "#576"}, {"href": "https://latest.datasette.io/-/plugins?all=1", "label": "https://latest.datasette.io/-/plugins?all=1"}, {"href": "https://github.com/simonw/datasette/issues/797", "label": "#797"}, {"href": "https://github.com/simonw/datasette/issues/802", "label": "#802"}, {"href": "https://github.com/simonw/datasette/issues/395", "label": "#395"}, {"href": "https://github.com/simonw/datasette/issues/777", "label": "#777"}, {"href": "https://github.com/simonw/datasette/issues/822", "label": "#822"}, {"href": "https://github.com/simonw/datasette/issues/804", "label": "#804"}, {"href": "https://github.com/simonw/datasette/issues/830", "label": "#830"}, {"href": "https://github.com/simonw/datasette/issues/837", "label": "#837"}] |
changelog:plugins-and-internals | changelog | plugins-and-internals | Plugins and internals | New plugin hook: filters_from_request(request, database, table, datasette) , which runs on the table page and can be used to support new custom query string parameters that modify the SQL query. ( #473 ) Added two additional methods for writing to the database: await db.execute_write_script(sql, block=True) and await db.execute_write_many(sql, params_seq, block=True) . ( #1570 ) The db.execute_write() internal method now defaults to blocking until the write operation has completed. Previously it defaulted to queuing the write and then continuing to run code while the write was in the queue. ( #1579 ) Database write connections now execute the prepare_connection(conn, database, datasette) plugin hook. ( #1564 ) The Datasette() constructor no longer requires the files= argument, and is now documented at Datasette class . ( #1563 ) The tracing feature now traces write queries, not just read queries. ( #1568 ) The query string variables exposed by request.args will now include blank strings for arguments such as foo in ?foo=&bar=1 rather than ignoring those parameters entirely. ( #1551 ) | ["Changelog", "0.60 (2022-01-13)"] | [{"href": "https://github.com/simonw/datasette/issues/473", "label": "#473"}, {"href": "https://github.com/simonw/datasette/issues/1570", "label": "#1570"}, {"href": "https://github.com/simonw/datasette/issues/1579", "label": "#1579"}, {"href": "https://github.com/simonw/datasette/issues/1564", "label": "#1564"}, {"href": "https://github.com/simonw/datasette/issues/1563", "label": "#1563"}, {"href": "https://github.com/simonw/datasette/issues/1568", "label": "#1568"}, {"href": "https://github.com/simonw/datasette/issues/1551", "label": "#1551"}] |
changelog:plugin-hooks | changelog | plugin-hooks | Plugin hooks | New plugin hook: handle_exception() , for custom handling of exceptions caught by Datasette. ( #1770 ) The render_cell() plugin hook is now also passed a row argument, representing the sqlite3.Row object that is being rendered. ( #1300 ) The configuration directory is now stored in datasette.config_dir , making it available to plugins. Thanks, Chris Amico. ( #1766 ) | ["Changelog", "0.62 (2022-08-14)"] | [{"href": "https://github.com/simonw/datasette/issues/1770", "label": "#1770"}, {"href": "https://github.com/simonw/datasette/issues/1300", "label": "#1300"}, {"href": "https://github.com/simonw/datasette/pull/1766", "label": "#1766"}] |
changelog:id90 | changelog | id90 | 0.25 (2018-09-19) | New plugin hooks, improved database view support and an easier way to use more recent versions of SQLite. New publish_subcommand plugin hook. A plugin can now add additional datasette publish publishers in addition to the default now and heroku , both of which have been refactored into default plugins. publish_subcommand documentation . Closes #349 New render_cell plugin hook. Plugins can now customize how values are displayed in the HTML tables produced by Datasette's browsable interface. datasette-json-html and datasette-render-images are two new plugins that use this hook. render_cell documentation . Closes #352 New extra_body_script plugin hook, enabling plugins to provide additional JavaScript that should be added to the page footer. extra_body_script documentation . extra_css_urls and extra_js_urls hooks now take additional optional parameters, allowing them to be more selective about which pages they apply to. Documentation . You can now use the sortable_columns metadata setting to explicitly enable sort-by-column in the interface for database views, as well as for specific tables. The new fts_table and fts_pk metadata settings can now be used to explicitly configure full-text search for a table or a view , even if that table is not directly coupled to the SQLite FTS feature in the database schema itself. Datasette will now use pysqlite3 in place of the standard library sqlite3 module if it has been installed in the current environment. This makes it much easier to run Datasette against a more recent version of SQLite, including the just-released SQLite 3.25.0 which adds wind… | ["Changelog"] | [{"href": "https://github.com/simonw/datasette/issues/349", "label": "#349"}, {"href": "https://github.com/simonw/datasette-json-html", "label": "datasette-json-html"}, {"href": "https://github.com/simonw/datasette-render-images", "label": "datasette-render-images"}, {"href": "https://github.com/simonw/datasette/issues/352", "label": "#352"}, {"href": "https://github.com/coleifer/pysqlite3", "label": "pysqlite3"}, {"href": "https://www.sqlite.org/releaselog/3_25_0.html", "label": "SQLite 3.25.0"}, {"href": "https://github.com/simonw/datasette/issues/360", "label": "#360"}] |
changelog:documentation | changelog | documentation | Documentation | New tutorial: Cleaning data with sqlite-utils and Datasette . Screenshots in the documentation are now maintained using shot-scraper , as described in Automating screenshots for the Datasette documentation using shot-scraper . ( #1844 ) More detailed command descriptions on the CLI reference page. ( #1787 ) New documentation on Running Datasette using OpenRC - thanks, Adam Simpson. ( #1825 ) | ["Changelog", "0.63 (2022-10-27)"] | [{"href": "https://datasette.io/tutorials/clean-data", "label": "Cleaning data with sqlite-utils and Datasette"}, {"href": "https://shot-scraper.datasette.io/", "label": "shot-scraper"}, {"href": "https://simonwillison.net/2022/Oct/14/automating-screenshots/", "label": "Automating screenshots for the Datasette documentation using shot-scraper"}, {"href": "https://github.com/simonw/datasette/issues/1844", "label": "#1844"}, {"href": "https://github.com/simonw/datasette/issues/1787", "label": "#1787"}, {"href": "https://github.com/simonw/datasette/pull/1825", "label": "#1825"}] |
settings:config-dir | settings | config-dir | Configuration directory mode | Normally you configure Datasette using command-line options. For a Datasette instance with custom templates, custom plugins, a static directory and several databases this can get quite verbose: $ datasette one.db two.db \ --metadata=metadata.json \ --template-dir=templates/ \ --plugins-dir=plugins \ --static css:css As an alternative to this, you can run Datasette in configuration directory mode. Create a directory with the following structure: # In a directory called my-app: my-app/one.db my-app/two.db my-app/metadata.json my-app/templates/index.html my-app/plugins/my_plugin.py my-app/static/my.css Now start Datasette by providing the path to that directory: $ datasette my-app/ Datasette will detect the files in that directory and automatically configure itself using them. It will serve all *.db files that it finds, will load metadata.json if it exists, and will load the templates , plugins and static folders if they are present. The files that can be included in this directory are as follows. All are optional. *.db (or *.sqlite3 or *.sqlite ) - SQLite database files that will be served by Datasette metadata.json - Metadata for those databases - metadata.yaml or metadata.yml can be used as well inspect-data.json - the result of running datasette inspect *.db --inspect-file=inspect-data.json from the configuration directory - any database files listed here will be treated as immutable, so they should not be changed while Datasette is running settings.json - settings that would normally be passed using --setting - here they should be stored as a JSON object of key/value pairs templates/ - a di… | ["Settings"] | [] |
changelog:features | changelog | features | Features | Now tested against Python 3.11. Docker containers used by datasette publish and datasette package both now use that version of Python. ( #1853 ) --load-extension option now supports entrypoints. Thanks, Alex Garcia. ( #1789 ) Facet size can now be set per-table with the new facet_size table metadata option. ( #1804 ) The truncate_cells_html setting now also affects long URLs in columns. ( #1805 ) The non-JavaScript SQL editor textarea now increases height to fit the SQL query. ( #1786 ) Facets are now displayed with better line-breaks in long values. Thanks, Daniel Rech. ( #1794 ) The settings.json file used in Configuration directory mode is now validated on startup. ( #1816 ) SQL queries can now include leading SQL comments, using /* ... */ or -- ... syntax. Thanks, Charles Nepote. ( #1860 ) SQL query is now re-displayed when terminated with a time limit error. ( #1819 ) The inspect data mechanism is now used to speed up server startup - thanks, Forest Gregg. ( #1834 ) In Configuration directory mode databases with filenames ending in .sqlite or .sqlite3 are now automatically added to the Datasette instance. ( #1646 ) Breadcrumb navigation display now respects the current user's permissions. ( #1831 ) | ["Changelog", "0.63 (2022-10-27)"] | [{"href": "https://github.com/simonw/datasette/issues/1853", "label": "#1853"}, {"href": "https://github.com/simonw/datasette/pull/1789", "label": "#1789"}, {"href": "https://github.com/simonw/datasette/issues/1804", "label": "#1804"}, {"href": "https://github.com/simonw/datasette/issues/1805", "label": "#1805"}, {"href": "https://github.com/simonw/datasette/issues/1786", "label": "#1786"}, {"href": "https://github.com/simonw/datasette/pull/1794", "label": "#1794"}, {"href": "https://github.com/simonw/datasette/issues/1816", "label": "#1816"}, {"href": "https://github.com/simonw/datasette/issues/1860", "label": "#1860"}, {"href": "https://github.com/simonw/datasette/issues/1819", "label": "#1819"}, {"href": "https://github.com/simonw/datasette/issues/1834", "label": "#1834"}, {"href": "https://github.com/simonw/datasette/issues/1646", "label": "#1646"}, {"href": "https://github.com/simonw/datasette/issues/1831", "label": "#1831"}] |
publish:cli-publish | publish | cli-publish | datasette publish | Once you have created a SQLite database (e.g. using csvs-to-sqlite ) you can deploy it to a hosting account using a single command. You will need a hosting account with Heroku or Google Cloud . Once you have created your account you will need to install and configure the heroku or gcloud command-line tools. | ["Publishing data"] | [{"href": "https://github.com/simonw/csvs-to-sqlite/", "label": "csvs-to-sqlite"}, {"href": "https://www.heroku.com/", "label": "Heroku"}, {"href": "https://cloud.google.com/", "label": "Google Cloud"}] |
contributing:contributing-running-tests | contributing | contributing-running-tests | Running the tests | Once you have done this, you can run the Datasette unit tests from inside your datasette/ directory using pytest like so: pytest You can run the tests faster using multiple CPU cores with pytest-xdist like this: pytest -n auto -m "not serial" -n auto detects the number of available cores automatically. The -m "not serial" skips tests that don't work well in a parallel test environment. You can run those tests separately like so: pytest -m "serial" | ["Contributing"] | [{"href": "https://docs.pytest.org/", "label": "pytest"}, {"href": "https://pypi.org/project/pytest-xdist/", "label": "pytest-xdist"}] |
deploying:deploying-openrc | deploying | deploying-openrc | Running Datasette using OpenRC | OpenRC is the service manager on non-systemd Linux distributions like Alpine Linux and Gentoo . Create an init script at /etc/init.d/datasette with the following contents: #!/sbin/openrc-run name="datasette" command="datasette" command_args="serve -h 0.0.0.0 /path/to/db.db" command_background=true pidfile="/run/${RC_SVCNAME}.pid" You then need to configure the service to run at boot and start it: rc-update add datasette rc-service datasette start | ["Deploying Datasette"] | [{"href": "https://www.alpinelinux.org/", "label": "Alpine Linux"}, {"href": "https://www.gentoo.org/", "label": "Gentoo"}] |
cli-reference:cli-help-plugins-help | cli-reference | cli-help-plugins-help | datasette plugins | Output JSON showing all currently installed plugins, their versions, whether they include static files or templates and which Plugin hooks they use. [[[cog help(["plugins", "--help"]) ]]] Usage: datasette plugins [OPTIONS] List currently installed plugins Options: --all Include built-in default plugins --plugins-dir DIRECTORY Path to directory containing custom plugins --help Show this message and exit. [[[end]]] Example output: [ { "name": "datasette-geojson", "static": false, "templates": false, "version": "0.3.1", "hooks": [ "register_output_renderer" ] }, { "name": "datasette-geojson-map", "static": true, "templates": false, "version": "0.4.0", "hooks": [ "extra_body_script", "extra_css_urls", "extra_js_urls" ] }, { "name": "datasette-leaflet", "static": true, "templates": false, "version": "0.2.2", "hooks": [ "extra_body_script", "extra_template_vars" ] } ] | ["CLI reference"] | [] |
cli-reference:cli-help-inspect-help | cli-reference | cli-help-inspect-help | datasette inspect | Outputs JSON representing introspected data about one or more SQLite database files. If you are opening an immutable database, you can pass this file to the --inspect-data option to improve Datasette's performance by allowing it to skip running row counts against the database when it first starts running: datasette inspect mydatabase.db > inspect-data.json datasette serve -i mydatabase.db --inspect-file inspect-data.json This performance optimization is used automatically by some of the datasette publish commands. You are unlikely to need to apply this optimization manually. [[[cog help(["inspect", "--help"]) ]]] Usage: datasette inspect [OPTIONS] [FILES]... Generate JSON summary of provided database files This can then be passed to "datasette --inspect-file" to speed up count operations against immutable database files. Options: --inspect-file TEXT --load-extension PATH:ENTRYPOINT? Path to a SQLite extension to load, and optional entrypoint --help Show this message and exit. [[[end]]] | ["CLI reference"] | [] |
cli-reference:cli-help-package-help | cli-reference | cli-help-package-help | datasette package | Package SQLite files into a Datasette Docker container, see datasette package . [[[cog help(["package", "--help"]) ]]] Usage: datasette package [OPTIONS] FILES... Package SQLite files into a Datasette Docker container Options: -t, --tag TEXT Name for the resulting Docker container, can optionally use name:tag format -m, --metadata FILENAME Path to JSON/YAML file containing metadata to publish --extra-options TEXT Extra options to pass to datasette serve --branch TEXT Install datasette from a GitHub branch e.g. main --template-dir DIRECTORY Path to directory containing custom templates --plugins-dir DIRECTORY Path to directory containing custom plugins --static MOUNT:DIRECTORY Serve static files from this directory at /MOUNT/... --install TEXT Additional packages (e.g. plugins) to install --spatialite Enable SpatialLite extension --version-note TEXT Additional note to show on /-/versions --secret TEXT Secret used for signing secure values, such as signed cookies -p, --port INTEGER RANGE Port to run the server on, defaults to 8001 [1<=x<=65535] --title TEXT Title for metadata --license TEXT License label for metadata --license_url TEXT License URL for metadata --source TEXT Source label for metadata --source_url TEXT Source URL for metadata --about TEXT About label for metadata --about_url TEXT About URL for metadata --help Show this message and exit. [[[end]]] | ["CLI reference"] | [] |
writing_plugins:writing-plugins-packaging | writing_plugins | writing-plugins-packaging | Packaging a plugin | Plugins can be packaged using Python setuptools. You can see an example of a packaged plugin at https://github.com/simonw/datasette-plugin-demos The example consists of two files: a setup.py file that defines the plugin: from setuptools import setup VERSION = "0.1" setup( name="datasette-plugin-demos", description="Examples of plugins for Datasette", author="Simon Willison", url="https://github.com/simonw/datasette-plugin-demos", license="Apache License, Version 2.0", version=VERSION, py_modules=["datasette_plugin_demos"], entry_points={ "datasette": [ "plugin_demos = datasette_plugin_demos" ] }, install_requires=["datasette"], ) And a Python module file, datasette_plugin_demos.py , that implements the plugin: from datasette import hookimpl import random @hookimpl def prepare_jinja2_environment(env): env.filters["uppercase"] = lambda u: u.upper() @hookimpl def prepare_connection(conn): conn.create_function( "random_integer", 2, random.randint ) Having built a plugin in this way you can turn it into an installable package using the following command: python3 setup.py sdist This will create a .tar.gz file in the dist/ directory. You can then install your new plugin into a Datasette virtual environment or Docker container using pip : pip install datasette-plugin-demos-0.1.tar.gz To learn how to upload your plugin to PyPI for use by other people, read the PyPA guide to Packaging and distributing projects . | ["Writing plugins"] | [{"href": "https://github.com/simonw/datasette-plugin-demos", "label": "https://github.com/simonw/datasette-plugin-demos"}, {"href": "https://pypi.org/", "label": "PyPI"}, {"href": "https://packaging.python.org/tutorials/distributing-packages/", "label": "Packaging and distributing projects"}] |
plugins:plugins-configuration | plugins | plugins-configuration | Plugin configuration | Plugins can have their own configuration, embedded in a Metadata file. Configuration options for plugins live within a "plugins" key in that file, which can be included at the root, database or table level. Here is an example of some plugin configuration for a specific table: { "databases": { "sf-trees": { "tables": { "Street_Tree_List": { "plugins": { "datasette-cluster-map": { "latitude_column": "lat", "longitude_column": "lng" } } } } } } } This tells the datasette-cluster-map column which latitude and longitude columns should be used for a table called Street_Tree_List inside a database file called sf-trees.db . | ["Plugins"] | [] |
internals:internals-datasette-client | internals | internals-datasette-client | datasette.client | Plugins can make internal simulated HTTP requests to the Datasette instance within which they are running. This ensures that all of Datasette's external JSON APIs are also available to plugins, while avoiding the overhead of making an external HTTP call to access those APIs. The datasette.client object is a wrapper around the HTTPX Python library , providing an async-friendly API that is similar to the widely used Requests library . It offers the following methods: await datasette.client.get(path, **kwargs) - returns HTTPX Response Execute an internal GET request against that path. await datasette.client.post(path, **kwargs) - returns HTTPX Response Execute an internal POST request. Use data={"name": "value"} to pass form parameters. await datasette.client.options(path, **kwargs) - returns HTTPX Response Execute an internal OPTIONS request. await datasette.client.head(path, **kwargs) - returns HTTPX Response Execute an internal HEAD request. await datasette.client.put(path, **kwargs) - returns HTTPX Response Execute an internal PUT request. await datasette.client.patch(path, **kwargs) - returns HTTPX Response … | ["Internals for plugins", "Datasette class"] | [{"href": "https://www.python-httpx.org/", "label": "HTTPX Python library"}, {"href": "https://requests.readthedocs.io/", "label": "Requests library"}, {"href": "https://www.python-httpx.org/async/", "label": "HTTPX Async documentation"}] |
changelog:register-routes-plugin-hooks | changelog | register-routes-plugin-hooks | register_routes() plugin hooks | Plugins can now register new views and routes via the register_routes(datasette) plugin hook ( #819 ). View functions can be defined that accept any of the current datasette object, the current request , or the ASGI scope , send and receive objects. | ["Changelog", "0.44 (2020-06-11)"] | [{"href": "https://github.com/simonw/datasette/issues/819", "label": "#819"}] |
changelog:cookie-methods | changelog | cookie-methods | Cookie methods | Plugins can now use the new response.set_cookie() method to set cookies. A new request.cookies method on the :ref:internals_request` can be used to read incoming cookies. | ["Changelog", "0.44 (2020-06-11)"] | [] |
changelog:secret-plugin-configuration-options | changelog | secret-plugin-configuration-options | Secret plugin configuration options | Plugins like datasette-auth-github need a safe way to set secret configuration options. Since the default mechanism for configuring plugins exposes those settings in /-/metadata a new mechanism was needed. Secret configuration values describes how plugins can now specify that their settings should be read from a file or an environment variable: { "plugins": { "datasette-auth-github": { "client_secret": { "$env": "GITHUB_CLIENT_SECRET" } } } } These plugin secrets can be set directly using datasette publish . See Custom metadata and plugins for details. ( #538 and #543 ) | ["Changelog", "0.29 (2019-07-07)"] | [{"href": "https://github.com/simonw/datasette-auth-github", "label": "datasette-auth-github"}, {"href": "https://github.com/simonw/datasette/issues/538", "label": "#538"}, {"href": "https://github.com/simonw/datasette/issues/543", "label": "#543"}] |
changelog:id66 | changelog | id66 | 0.37 (2020-02-25) | Plugins now have a supported mechanism for writing to a database, using the new .execute_write() and .execute_write_fn() methods. Documentation . ( #682 ) Immutable databases that have had their rows counted using the inspect command now use the calculated count more effectively - thanks, Kevin Keogh. ( #666 ) --reload no longer restarts the server if a database file is modified, unless that database was opened immutable mode with -i . ( #494 ) New ?_searchmode=raw option turns off escaping for FTS queries in ?_search= allowing full use of SQLite's FTS5 query syntax . ( #676 ) | ["Changelog"] | [{"href": "https://github.com/simonw/datasette/issues/682", "label": "#682"}, {"href": "https://github.com/simonw/datasette/pull/666", "label": "#666"}, {"href": "https://github.com/simonw/datasette/issues/494", "label": "#494"}, {"href": "https://www.sqlite.org/fts5.html#full_text_query_syntax", "label": "FTS5 query syntax"}, {"href": "https://github.com/simonw/datasette/issues/676", "label": "#676"}] |
writing_plugins:writing-plugins-cookiecutter | writing_plugins | writing-plugins-cookiecutter | Starting an installable plugin using cookiecutter | Plugins that can be installed should be written as Python packages using a setup.py file. The quickest way to start writing one an installable plugin is to use the datasette-plugin cookiecutter template. This creates a new plugin structure for you complete with an example test and GitHub Actions workflows for testing and publishing your plugin. Install cookiecutter and then run this command to start building a plugin using the template: cookiecutter gh:simonw/datasette-plugin Read a cookiecutter template for writing Datasette plugins for more information about this template. | ["Writing plugins"] | [{"href": "https://github.com/simonw/datasette-plugin", "label": "datasette-plugin"}, {"href": "https://cookiecutter.readthedocs.io/en/stable/installation.html", "label": "Install cookiecutter"}, {"href": "https://simonwillison.net/2020/Jun/20/cookiecutter-plugins/", "label": "a cookiecutter template for writing Datasette plugins"}] |
writing_plugins:writing-plugins-building-urls | writing_plugins | writing-plugins-building-urls | Building URLs within plugins | Plugins that define their own custom user interface elements may need to link to other pages within Datasette. This can be a bit tricky if the Datasette instance is using the base_url configuration setting to run behind a proxy, since that can cause Datasette's URLs to include an additional prefix. The datasette.urls object provides internal methods for correctly generating URLs to different pages within Datasette, taking any base_url configuration into account. This object is exposed in templates as the urls variable, which can be used like this: Back to the <a href="{{ urls.instance() }}">Homepage</a> See datasette.urls for full details on this object. | ["Writing plugins"] | [] |
authentication:authentication-actor-matches-allow | authentication | authentication-actor-matches-allow | actor_matches_allow() | Plugins that wish to implement this same "allow" block permissions scheme can take advantage of the datasette.utils.actor_matches_allow(actor, allow) function: from datasette.utils import actor_matches_allow actor_matches_allow({"id": "root"}, {"id": "*"}) # returns True The currently authenticated actor is made available to plugins as request.actor . | ["Authentication and permissions"] | [] |
changelog:authentication | changelog | authentication | Authentication | Prior to this release the Datasette ecosystem has treated authentication as exclusively the realm of plugins, most notably through datasette-auth-github . 0.44 introduces Authentication and permissions as core Datasette concepts ( #699 ). This enables different plugins to share responsibility for authenticating requests - you might have one plugin that handles user accounts and another one that allows automated access via API keys, for example. You'll need to install plugins if you want full user accounts, but default Datasette can now authenticate a single root user with the new --root command-line option, which outputs a one-time use URL to authenticate as a root actor ( #784 ): $ datasette fixtures.db --root http://127.0.0.1:8001/-/auth-token?token=5b632f8cd44b868df625f5a6e2185d88eea5b22237fd3cc8773f107cc4fd6477 INFO: Started server process [14973] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit) Plugins can implement new ways of authenticating users using the new actor_from_request(datasette, request) hook. | ["Changelog", "0.44 (2020-06-11)"] | [{"href": "https://github.com/simonw/datasette-auth-github", "label": "datasette-auth-github"}, {"href": "https://github.com/simonw/datasette/issues/699", "label": "#699"}, {"href": "https://github.com/simonw/datasette/issues/784", "label": "#784"}] |
internals:datasette-databases | internals | datasette-databases | .databases | Property exposing a collections.OrderedDict of databases currently connected to Datasette. The dictionary keys are the name of the database that is used in the URL - e.g. /fixtures would have a key of "fixtures" . The values are Database class instances. All databases are listed, irrespective of user permissions. This means that the _internal database will always be listed here. | ["Internals for plugins", "Datasette class"] | [] |
testing_plugins:testing-plugins-fixtures | testing_plugins | testing-plugins-fixtures | Using pytest fixtures | Pytest fixtures can be used to create initial testable objects which can then be used by multiple tests. A common pattern for Datasette plugins is to create a fixture which sets up a temporary test database and wraps it in a Datasette instance. Here's an example that uses the sqlite-utils library to populate a temporary test database. It also sets the title of that table using a simulated metadata.json configuration: from datasette.app import Datasette import pytest import sqlite_utils @pytest.fixture(scope="session") def datasette(tmp_path_factory): db_directory = tmp_path_factory.mktemp("dbs") db_path = db_directory / "test.db" db = sqlite_utils.Database(db_path) db["dogs"].insert_all( [ {"id": 1, "name": "Cleo", "age": 5}, {"id": 2, "name": "Pancakes", "age": 4}, ], pk="id", ) datasette = Datasette( [db_path], metadata={ "databases": { "test": { "tables": { "dogs": {"title": "Some dogs"} } } } }, ) return datasette @pytest.mark.asyncio async def test_example_table_json(datasette): response = await datasette.client.get( "/test/dogs.json?_shape=array" ) assert response.status_code == 200 assert response.json() == [ {"id": 1, "name": "Cleo", "age": 5}, {"id": 2, "name": "Pancakes", "age": 4}, ] @pytest.mark.asyncio async def test_example_table_html(datasette): response = await datasette.client.get("/test/dogs") assert ">Some dogs</h1>" in response.text Here the datasette() function defines the fixture, which is than automatically passed to the two test functions based on pytest automatically matching their datasette function parameters. The @pytest.fixture(scope="session") line here ensures the fixture is reused for the full pytest execution session. This… | ["Testing plugins"] | [{"href": "https://docs.pytest.org/en/stable/fixture.html", "label": "Pytest fixtures"}, {"href": "https://sqlite-utils.datasette.io/en/stable/python-api.html", "label": "sqlite-utils library"}] |
changelog:id151 | changelog | id151 | 0.17 (2018-04-13) | Release 0.17 to fix issues with PyPI | ["Changelog"] | [] |
plugin_hooks:plugin-register-facet-classes | plugin_hooks | plugin-register-facet-classes | register_facet_classes() | Return a list of additional Facet subclasses to be registered. The design of this plugin hook is unstable and may change. See issue 830 . Each Facet subclass implements a new type of facet operation. The class should look like this: class SpecialFacet(Facet): # This key must be unique across all facet classes: type = "special" async def suggest(self): # Use self.sql and self.params to suggest some facets suggested_facets = [] suggested_facets.append( { "name": column, # Or other unique name # Construct the URL that will enable this facet: "toggle_url": self.ds.absolute_url( self.request, path_with_added_args( self.request, {"_facet": column} ), ), } ) return suggested_facets async def facet_results(self): # This should execute the facet operation and return results, again # using self.sql and self.params as the starting point facet_results = [] facets_timed_out = [] facet_size = self.get_facet_size() # Do some calculations here... for column in columns_selected_for_facet: try: facet_results_values = [] # More calculations... facet_results_values.append( { "value": value, "label": label, "count": count, "toggle_url": self.ds.absolute_url( self.request, toggle_path ), "selected": selected, } ) facet_results.append( { "name": column, "results": facet_results_values, "trunc… | ["Plugin hooks"] | [{"href": "https://github.com/simonw/datasette/issues/830", "label": "issue 830"}, {"href": "https://github.com/simonw/datasette/blob/main/datasette/facets.py", "label": "datasette/facets.py"}] |
plugin_hooks:plugin-asgi-wrapper | plugin_hooks | plugin-asgi-wrapper | asgi_wrapper(datasette) | Return an ASGI middleware wrapper function that will be applied to the Datasette ASGI application. This is a very powerful hook. You can use it to manipulate the entire Datasette response, or even to configure new URL routes that will be handled by your own custom code. You can write your ASGI code directly against the low-level specification, or you can use the middleware utilities provided by an ASGI framework such as Starlette . This example plugin adds a x-databases HTTP header listing the currently attached databases: from datasette import hookimpl from functools import wraps @hookimpl def asgi_wrapper(datasette): def wrap_with_databases_header(app): @wraps(app) async def add_x_databases_header( scope, receive, send ): async def wrapped_send(event): if event["type"] == "http.response.start": original_headers = ( event.get("headers") or [] ) event = { "type": event["type"], "status": event["status"], "headers": original_headers + [ [ b"x-databases", ", ".join( datasette.databases.keys() ).encode("utf-8"), ] ], } await send(event) await app(scope, receive, wrapped_send) return add_x_databases_header return wrap_with_databases_header Examples: datasette-cors , datasette-pyinstrument , datasette-total-page-time | ["Plugin hooks"] | [{"href": "https://asgi.readthedocs.io/", "label": "ASGI"}, {"href": "https://www.starlette.io/middleware/", "label": "Starlette"}, {"href": "https://datasette.io/plugins/datasette-cors", "label": "datasette-cors"}, {"href": "https://datasette.io/plugins/datasette-pyinstrument", "label": "datasette-pyinstrument"}, {"href": "https://datasette.io/plugins/datasette-total-page-time", "label": "datasette-total-page-time"}] |
cli-reference:cli-help-help | cli-reference | cli-help-help | datasette --help | Running datasette --help shows a list of all of the available commands. [[[cog help(["--help"]) ]]] Usage: datasette [OPTIONS] COMMAND [ARGS]... Datasette is an open source multi-tool for exploring and publishing data About Datasette: https://datasette.io/ Full documentation: https://docs.datasette.io/ Options: --version Show the version and exit. --help Show this message and exit. Commands: serve* Serve up specified SQLite database files with a web UI inspect Generate JSON summary of provided database files install Install plugins and packages from PyPI into the same... package Package SQLite files into a Datasette Docker container plugins List currently installed plugins publish Publish specified SQLite database files to the internet along... uninstall Uninstall plugins and Python packages from the Datasette... [[[end]]] Additional commands added by plugins that use the register_commands(cli) hook will be listed here as well. | ["CLI reference"] | [] |
internals:internals-tracer | internals | internals-tracer | datasette.tracer | Running Datasette with --setting trace_debug 1 enables trace debug output, which can then be viewed by adding ?_trace=1 to the query string for any page. You can see an example of this at the bottom of latest.datasette.io/fixtures/facetable?_trace=1 . The JSON output shows full details of every SQL query that was executed to generate the page. The datasette-pretty-traces plugin can be installed to provide a more readable display of this information. You can see a demo of that here . You can add your own custom traces to the JSON output using the trace() context manager. This takes a string that identifies the type of trace being recorded, and records any keyword arguments as additional JSON keys on the resulting trace object. The start and end time, duration and a traceback of where the trace was executed will be automatically attached to the JSON object. This example uses trace to record the start, end and duration of any HTTP GET requests made using the function: from datasette.tracer import trace import httpx async def fetch_url(url): with trace("fetch-url", url=url): async with httpx.AsyncClient() as client: return await client.get(url) | ["Internals for plugins"] | [{"href": "https://latest.datasette.io/fixtures/facetable?_trace=1", "label": "latest.datasette.io/fixtures/facetable?_trace=1"}, {"href": "https://datasette.io/plugins/datasette-pretty-traces", "label": "datasette-pretty-traces"}, {"href": "https://latest-with-plugins.datasette.io/github/commits?_trace=1", "label": "a demo of that here"}] |
changelog:v0-28-register-output-renderer | changelog | v0-28-register-output-renderer | register_output_renderer plugins | Russ Garrett implemented a new Datasette plugin hook called register_output_renderer ( #441 ) which allows plugins to create additional output renderers in addition to Datasette's default .json and .csv . Russ's in-development datasette-geo plugin includes an example of this hook being used to output .geojson automatically converted from SpatiaLite. | ["Changelog", "0.28 (2019-05-19)"] | [{"href": "https://github.com/simonw/datasette/pull/441", "label": "#441"}, {"href": "https://github.com/russss/datasette-geo", "label": "datasette-geo"}, {"href": "https://github.com/russss/datasette-geo/blob/d4cecc020848bbde91e9e17bf352f7c70bc3dccf/datasette_plugin_geo/geojson.py", "label": "an example"}] |
full_text_search:full-text-search-advanced-queries | full_text_search | full-text-search-advanced-queries | Advanced SQLite search queries | SQLite full-text search includes support for a variety of advanced queries , including AND , OR , NOT and NEAR . By default Datasette disables these features to ensure they do not cause errors or confusion for users who are not aware of them. You can disable this escaping and use the advanced queries by adding &_searchmode=raw to the table page query string. If you want to enable these operators by default for a specific table, you can do so by adding "searchmode": "raw" to the metadata configuration for that table, see Configuring full-text search for a table or view . If that option has been specified in the table metadata but you want to over-ride it and return to the default behavior you can append &_searchmode=escaped to the query string. | ["Full-text search"] | [{"href": "https://www.sqlite.org/fts5.html#full_text_query_syntax", "label": "a variety of advanced queries"}] |
sql_queries:id3 | sql_queries | id3 | Cross-database queries | SQLite has the ability to run queries that join across multiple databases. Up to ten databases can be attached to a single SQLite connection and queried together. Datasette can execute joins across multiple databases if it is started with the --crossdb option: datasette fixtures.db extra_database.db --crossdb If it is started in this way, the /_memory page can be used to execute queries that join across multiple databases. References to tables in attached databases should be preceded by the database name and a period. For example, this query will show a list of tables across both of the above databases: select 'fixtures' as database, * from [fixtures].sqlite_master union select 'extra_database' as database, * from [extra_database].sqlite_master Try that out here . | ["Running SQL queries"] | [{"href": "https://latest.datasette.io/_memory?sql=select%0D%0A++%27fixtures%27+as+database%2C+*%0D%0Afrom%0D%0A++%5Bfixtures%5D.sqlite_master%0D%0Aunion%0D%0Aselect%0D%0A++%27extra_database%27+as+database%2C+*%0D%0Afrom%0D%0A++%5Bextra_database%5D.sqlite_master", "label": "Try that out here"}] |
full_text_search:id1 | full_text_search | id1 | Full-text search | SQLite includes a powerful mechanism for enabling full-text search against SQLite records. Datasette can detect if a table has had full-text search configured for it in the underlying database and display a search interface for filtering that table. Here's an example search : Datasette automatically detects which tables have been configured for full-text search. | [] | [{"href": "https://www.sqlite.org/fts3.html", "label": "a powerful mechanism for enabling full-text search"}, {"href": "https://register-of-members-interests.datasettes.com/regmem/items?_search=hamper&_sort_desc=date", "label": "an example search"}] |
internals:database-execute-write | internals | database-execute-write | await db.execute_write(sql, params=None, block=True) | SQLite only allows one database connection to write at a time. Datasette handles this for you by maintaining a queue of writes to be executed against a given database. Plugins can submit write operations to this queue and they will be executed in the order in which they are received. This method can be used to queue up a non-SELECT SQL query to be executed against a single write connection to the database. You can pass additional SQL parameters as a tuple or dictionary. The method will block until the operation is completed, and the return value will be the return from calling conn.execute(...) using the underlying sqlite3 Python library. If you pass block=False this behaviour changes to "fire and forget" - queries will be added to the write queue and executed in a separate thread while your code can continue to do other things. The method will return a UUID representing the queued task. | ["Internals for plugins", "Database class"] | [] |
installation:installation-extensions | installation | installation-extensions | A note about extensions | SQLite supports extensions, such as SpatiaLite for geospatial operations. These can be loaded using the --load-extension argument, like so: datasette --load-extension=/usr/local/lib/mod_spatialite.dylib Some Python installations do not include support for SQLite extensions. If this is the case you will see the following error when you attempt to load an extension: Your Python installation does not have the ability to load SQLite extensions. In some cases you may see the following error message instead: AttributeError: 'sqlite3.Connection' object has no attribute 'enable_load_extension' On macOS the easiest fix for this is to install Datasette using Homebrew: brew install datasette Use which datasette to confirm that datasette will run that version. The output should look something like this: /usr/local/opt/datasette/bin/datasette If you get a different location here such as /Library/Frameworks/Python.framework/Versions/3.10/bin/datasette you can run the following command to cause datasette to execute the Homebrew version instead: alias datasette=$(echo $(brew --prefix datasette)/bin/datasette) You can undo this operation using: unalias datasette If you need to run SQLite with extension support for other Python code, you can do so by install Python itself using Homebrew: brew install python Then executing Python using: /usr/local/opt/python@3/libexec/bin/python A more convenient way to work with this version of Python may be to use it to create a virtual environment: /usr/local/opt/python@3/libexec/bin/python -m venv datasette-venv Then activate it like this: source datasette-venv/bin/activate Now running python and pip will work against a version of … | ["Installation"] | [] |
binary_data:binary | binary_data | binary | Binary data | SQLite tables can contain binary data in BLOB columns. Datasette includes special handling for these binary values. The Datasette interface detects binary values and provides a link to download their content, for example on https://latest.datasette.io/fixtures/binary_data Binary data is represented in .json exports using Base64 encoding. https://latest.datasette.io/fixtures/binary_data.json?_shape=array [ { "rowid": 1, "data": { "$base64": true, "encoded": "FRwCx60F/g==" } }, { "rowid": 2, "data": { "$base64": true, "encoded": "FRwDx60F/g==" } }, { "rowid": 3, "data": null } ] | [] | [{"href": "https://latest.datasette.io/fixtures/binary_data", "label": "https://latest.datasette.io/fixtures/binary_data"}, {"href": "https://latest.datasette.io/fixtures/binary_data.json?_shape=array", "label": "https://latest.datasette.io/fixtures/binary_data.json?_shape=array"}] |
changelog:binary-data | changelog | binary-data | Binary data | SQLite tables can contain binary data in BLOB columns. Datasette now provides links for users to download this data directly from Datasette, and uses those links to make binary data available from CSV exports. See Binary data for more details. ( #1036 and #1034 ). | ["Changelog", "0.51 (2020-10-31)"] | [{"href": "https://github.com/simonw/datasette/issues/1036", "label": "#1036"}, {"href": "https://github.com/simonw/datasette/issues/1034", "label": "#1034"}] |
changelog:id180 | changelog | id180 | 0.13 (2017-11-24) | Search now applies to current filters. Combined search into the same form as filters. Closes #133 Much tidier design for table view header. Closes #147 Added ?column__not=blah filter. Closes #148 Row page now resolves foreign keys. Closes #132 Further tweaks to select/input filter styling. Refs #86 - thanks for the help, @natbat! Show linked foreign key in table cells. Added UI for editing table filters. Refs #86 Hide FTS-created tables on index pages. Closes #129 Add publish to heroku support [Jacob Kaplan-Moss] datasette publish heroku mydb.db Pull request #104 Initial implementation of ?_group_count=column . URL shortcut for counting rows grouped by one or more columns. ?_group_count=column1&_group_count=column2 works as well. SQL generated looks like this: select "qSpecies", count(*) as "count" from Street_Tree_List group by "qSpecies" order by "count" desc limit 100 Or for two columns like this: select "qSpecies", "qSiteInfo", count(*) as "count" from Street_Tree_List group by "qSpecies", "qSiteInfo" order by "count" desc limit 100 Refs #44 Added --build=mas… | ["Changelog"] | [{"href": "https://github.com/simonw/datasette/issues/133", "label": "#133"}, {"href": "https://github.com/simonw/datasette/issues/147", "label": "#147"}, {"href": "https://github.com/simonw/datasette/issues/148", "label": "#148"}, {"href": "https://github.com/simonw/datasette/issues/132", "label": "#132"}, {"href": "https://github.com/simonw/datasette/issues/86", "label": "#86"}, {"href": "https://github.com/simonw/datasette/issues/86", "label": "#86"}, {"href": "https://github.com/simonw/datasette/issues/129", "label": "#129"}, {"href": "https://github.com/simonw/datasette/issues/104", "label": "#104"}, {"href": "https://github.com/simonw/datasette/issues/44", "label": "#44"}, {"href": "https://github.com/simonw/datasette/issues/131", "label": "#131"}, {"href": "https://github.com/simonw/datasette/issues/117", "label": "#117"}, {"href": "https://github.com/simonw/datasette/issues/115", "label": "#115"}, {"href": "https://github.com/simonw/datasette/issues/115", "label": "#115"}, {"href": "https://github.com/simonw/datasette/issues/107", "label": "#107"}, {"href": "https://github.com/simonw/datasette/issues/114", "label": "#114"}, {"href": "https://github.com/simonw/datasette/issues/110", "label": "#110"}] |
changelog:id12 | changelog | id12 | 0.63 (2022-10-27) | See Datasette 0.63: The annotated release notes for more background on the changes in this release. | ["Changelog"] | [{"href": "https://simonwillison.net/2022/Oct/27/datasette-0-63/", "label": "Datasette 0.63: The annotated release notes"}] |
cli-reference:cli-help-publish-cloudrun-help | cli-reference | cli-help-publish-cloudrun-help | datasette publish cloudrun | See Publishing to Google Cloud Run . [[[cog help(["publish", "cloudrun", "--help"]) ]]] Usage: datasette publish cloudrun [OPTIONS] [FILES]... Publish databases to Datasette running on Cloud Run Options: -m, --metadata FILENAME Path to JSON/YAML file containing metadata to publish --extra-options TEXT Extra options to pass to datasette serve --branch TEXT Install datasette from a GitHub branch e.g. main --template-dir DIRECTORY Path to directory containing custom templates --plugins-dir DIRECTORY Path to directory containing custom plugins --static MOUNT:DIRECTORY Serve static files from this directory at /MOUNT/... --install TEXT Additional packages (e.g. plugins) to install --plugin-secret <TEXT TEXT TEXT>... Secrets to pass to plugins, e.g. --plugin- secret datasette-auth-github client_id xxx --version-note TEXT Additional note to show on /-/versions --secret TEXT Secret used for signing secure values, such as signed cookies --title TEXT Title for metadata --license TEXT License label for metadata --license_url TEXT License URL for metadata --source TEXT Source label for metadata --source_url TEXT Source URL for metadata --about TEXT About label for metadata --about_url TEXT About URL for metadata -n, --name TEXT Application name to use when building --service TEXT Cloud Run service to deploy (or over-write) --spatialite Enable SpatialLite extension --show-files Output the generated Dockerfile and metad… | ["CLI reference"] | [] |
cli-reference:cli-help-publish-heroku-help | cli-reference | cli-help-publish-heroku-help | datasette publish heroku | See Publishing to Heroku . [[[cog help(["publish", "heroku", "--help"]) ]]] Usage: datasette publish heroku [OPTIONS] [FILES]... Publish databases to Datasette running on Heroku Options: -m, --metadata FILENAME Path to JSON/YAML file containing metadata to publish --extra-options TEXT Extra options to pass to datasette serve --branch TEXT Install datasette from a GitHub branch e.g. main --template-dir DIRECTORY Path to directory containing custom templates --plugins-dir DIRECTORY Path to directory containing custom plugins --static MOUNT:DIRECTORY Serve static files from this directory at /MOUNT/... --install TEXT Additional packages (e.g. plugins) to install --plugin-secret <TEXT TEXT TEXT>... Secrets to pass to plugins, e.g. --plugin- secret datasette-auth-github client_id xxx --version-note TEXT Additional note to show on /-/versions --secret TEXT Secret used for signing secure values, such as signed cookies --title TEXT Title for metadata --license TEXT License label for metadata --license_url TEXT License URL for metadata --source TEXT Source label for metadata --source_url TEXT Source URL for metadata --about TEXT About label for metadata --about_url TEXT About URL for metadata -n, --name TEXT Application name to use when deploying --tar TEXT --tar option to pass to Heroku, e.g. --tar=/usr/local/bin/gtar --generate-dir DIRECTORY Output generated application files and stop without deploying --h… | ["CLI reference"] | [] |
changelog:id57 | changelog | id57 | 0.44 (2020-06-11) | See also Datasette 0.44: The annotated release notes . Authentication and permissions, writable canned queries, flash messages, new plugin hooks and more. | ["Changelog"] | [{"href": "https://simonwillison.net/2020/Jun/12/annotated-release-notes/", "label": "Datasette 0.44: The annotated release notes"}] |
changelog:id55 | changelog | id55 | 0.45 (2020-07-01) | See also Datasette 0.45: The annotated release notes . Magic parameters for canned queries, a log out feature, improved plugin documentation and four new plugin hooks. | ["Changelog"] | [{"href": "https://simonwillison.net/2020/Jul/1/datasette-045/", "label": "Datasette 0.45: The annotated release notes"}] |
changelog:id48 | changelog | id48 | 0.49 (2020-09-14) | See also Datasette 0.49: The annotated release notes . Writable canned queries now expose a JSON API, see JSON API for writable canned queries . ( #880 ) New mechanism for defining page templates with custom path parameters - a template file called pages/about/{slug}.html will be used to render any requests to /about/something . See Path parameters for pages . ( #944 ) register_output_renderer() render functions can now return a Response . ( #953 ) New --upgrade option for datasette install . ( #945 ) New datasette --pdb option. ( #962 ) datasette --get exit code now reflects the internal HTTP status code. ( #947 ) New raise_404() template function for returning 404 errors. ( #964 ) datasette publish heroku now deploys using Python 3.8.5 Upgraded CodeMirror to 5.57.0. ( #948 ) Upgraded code style to Black 20.8b1. ( #958 ) Fixed bug where selected facets were not correctly persisted in hidden form fields on the table page. ( #963 ) Renamed the default error template from 500.html to error.html . Custom error pages are now documented, see Custom error pages . ( #965 ) | ["Changelog"] | [{"href": "https://simonwillison.net/2020/Sep/15/datasette-0-49/", "label": "Datasette 0.49: The annotated release notes"}, {"href": "https://github.com/simonw/datasette/issues/880", "label": "#880"}, {"href": "https://github.com/simonw/datasette/issues/944", "label": "#944"}, {"href": "https://github.com/simonw/datasette/issues/953", "label": "#953"}, {"href": "https://github.com/simonw/datasette/issues/945", "label": "#945"}, {"href": "https://github.com/simonw/datasette/issues/962", "label": "#962"}, {"href": "https://github.com/simonw/datasette/issues/947", "label": "#947"}, {"href": "https://github.com/simonw/datasette/issues/964", "label": "#964"}, {"href": "https://codemirror.net/", "label": "CodeMirror"}, {"href": "https://github.com/simonw/datasette/issues/948", "label": "#948"}, {"href": "https://github.com/simonw/datasette/issues/958", "label": "#958"}, {"href": "https://github.com/simonw/datasette/issues/963", "label": "#963"}, {"href": "https://github.com/simonw/datasette/issues/965", "label": "#965"}] |
settings:setting-cache-size-kb | settings | setting-cache-size-kb | cache_size_kb | Sets the amount of memory SQLite uses for its per-connection cache , in KB. datasette mydatabase.db --setting cache_size_kb 5000 | ["Settings", "Settings"] | [{"href": "https://www.sqlite.org/pragma.html#pragma_cache_size", "label": "per-connection cache"}] |
binary_data:binary-plugins | binary_data | binary-plugins | Binary plugins | Several Datasette plugins are available that change the way Datasette treats binary data. datasette-render-binary modifies Datasette's default interface to show an automatic guess at what type of binary data is being stored, along with a visual representation of the binary value that displays ASCII strings directly in the interface. datasette-render-images detects common image formats and renders them as images directly in the Datasette interface. datasette-media allows Datasette interfaces to be configured to serve binary files from configured SQL queries, and includes the ability to resize images directly before serving them. | ["Binary data"] | [{"href": "https://github.com/simonw/datasette-render-binary", "label": "datasette-render-binary"}, {"href": "https://github.com/simonw/datasette-render-images", "label": "datasette-render-images"}, {"href": "https://github.com/simonw/datasette-media", "label": "datasette-media"}] |
settings:setting-suggest-facets | settings | setting-suggest-facets | suggest_facets | Should Datasette calculate suggested facets? On by default, turn this off like so: datasette mydatabase.db --setting suggest_facets off | ["Settings", "Settings"] | [] |
settings:setting-allow-download | settings | setting-allow-download | allow_download | Should users be able to download the original SQLite database using a link on the database index page? This is turned on by default. However, databases can only be downloaded if they are served in immutable mode and not in-memory. If downloading is unavailable for either of these reasons, the download link is hidden even if allow_download is on. To disable database downloads, use the following: datasette mydatabase.db --setting allow_download off | ["Settings", "Settings"] | [] |
settings:setting-default-allow-sql | settings | setting-default-allow-sql | default_allow_sql | Should users be able to execute arbitrary SQL queries by default? Setting this to off causes permission checks for execute-sql to fail by default. datasette mydatabase.db --setting default_allow_sql off There are two ways to achieve this: the other is to add "allow_sql": false to your metadata.json file, as described in Controlling the ability to execute arbitrary SQL . This setting offers a more convenient way to do this. | ["Settings", "Settings"] | [] |
changelog:id37 | changelog | id37 | 0.52.4 (2020-12-05) | Show pysqlite3 version on /-/versions , if installed. ( #1125 ) Errors output by Datasette (e.g. for invalid SQL queries) now go to stderr , not stdout . ( #1131 ) Fix for a startup error on windows caused by unnecessary from os import EX_CANTCREAT - thanks, Abdussamet Koçak. ( #1094 ) | ["Changelog"] | [{"href": "https://github.com/coleifer/pysqlite3", "label": "pysqlite3"}, {"href": "https://github.com/simonw/datasette/issues/1125", "label": "#1125"}, {"href": "https://github.com/simonw/datasette/issues/1131", "label": "#1131"}, {"href": "https://github.com/simonw/datasette/issues/1094", "label": "#1094"}] |
cli-reference:cli-help-publish-help | cli-reference | cli-help-publish-help | datasette publish | Shows a list of available deployment targets for publishing data with Datasette. Additional deployment targets can be added by plugins that use the publish_subcommand(publish) hook. [[[cog help(["publish", "--help"]) ]]] Usage: datasette publish [OPTIONS] COMMAND [ARGS]... Publish specified SQLite database files to the internet along with a Datasette-powered interface and API Options: --help Show this message and exit. Commands: cloudrun Publish databases to Datasette running on Cloud Run heroku Publish databases to Datasette running on Heroku [[[end]]] | ["CLI reference"] | [] |
introspection:jsondataview-plugins | introspection | jsondataview-plugins | /-/plugins | Shows a list of currently installed plugins and their versions. Plugins example : [ { "name": "datasette_cluster_map", "static": true, "templates": false, "version": "0.10", "hooks": ["extra_css_urls", "extra_js_urls", "extra_body_script"] } ] Add ?all=1 to include details of the default plugins baked into Datasette. | ["Introspection"] | [{"href": "https://san-francisco.datasettes.com/-/plugins", "label": "Plugins example"}] |
introspection:jsondataview-databases | introspection | jsondataview-databases | /-/databases | Shows currently attached databases. Databases example : [ { "hash": null, "is_memory": false, "is_mutable": true, "name": "fixtures", "path": "fixtures.db", "size": 225280 } ] | ["Introspection"] | [{"href": "https://latest.datasette.io/-/databases", "label": "Databases example"}] |
introspection:jsondataview-threads | introspection | jsondataview-threads | /-/threads | Shows details of threads and asyncio tasks. Threads example : { "num_threads": 2, "threads": [ { "daemon": false, "ident": 4759197120, "name": "MainThread" }, { "daemon": true, "ident": 123145319682048, "name": "Thread-1" }, ], "num_tasks": 3, "tasks": [ "<Task pending coro=<RequestResponseCycle.run_asgi() running at uvicorn/protocols/http/httptools_impl.py:385> cb=[set.discard()]>", "<Task pending coro=<Server.serve() running at uvicorn/main.py:361> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x10365c3d0>()]> cb=[run_until_complete.<locals>.<lambda>()]>", "<Task pending coro=<LifespanOn.main() running at uvicorn/lifespan/on.py:48> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x10364f050>()]>>" ] } | ["Introspection"] | [{"href": "https://latest.datasette.io/-/threads", "label": "Threads example"}] |
introspection:jsondataview-config | introspection | jsondataview-config | /-/settings | Shows the Settings for this instance of Datasette. Settings example : { "default_facet_size": 30, "default_page_size": 100, "facet_suggest_time_limit_ms": 50, "facet_time_limit_ms": 1000, "max_returned_rows": 1000, "sql_time_limit_ms": 1000 } | ["Introspection"] | [{"href": "https://fivethirtyeight.datasettes.com/-/settings", "label": "Settings example"}] |
introspection:jsondataview-metadata | introspection | jsondataview-metadata | /-/metadata | Shows the contents of the metadata.json file that was passed to datasette serve , if any. Metadata example : { "license": "CC Attribution 4.0 License", "license_url": "http://creativecommons.org/licenses/by/4.0/", "source": "fivethirtyeight/data on GitHub", "source_url": "https://github.com/fivethirtyeight/data", "title": "Five Thirty Eight", "databases": { } } | ["Introspection"] | [{"href": "https://fivethirtyeight.datasettes.com/-/metadata", "label": "Metadata example"}] |
introspection:jsondataview-actor | introspection | jsondataview-actor | /-/actor | Shows the currently authenticated actor. Useful for debugging Datasette authentication plugins. { "actor": { "id": 1, "username": "some-user" } } | ["Introspection"] | [] |
introspection:jsondataview-versions | introspection | jsondataview-versions | /-/versions | Shows the version of Datasette, Python and SQLite. Versions example : { "datasette": { "version": "0.60" }, "python": { "full": "3.8.12 (default, Dec 21 2021, 10:45:09) \n[GCC 10.2.1 20210110]", "version": "3.8.12" }, "sqlite": { "extensions": { "json1": null }, "fts_versions": [ "FTS5", "FTS4", "FTS3" ], "compile_options": [ "COMPILER=gcc-6.3.0 20170516", "ENABLE_FTS3", "ENABLE_FTS4", "ENABLE_FTS5", "ENABLE_JSON1", "ENABLE_RTREE", "THREADSAFE=1" ], "version": "3.37.0" } } | ["Introspection"] | [{"href": "https://latest.datasette.io/-/versions", "label": "Versions example"}] |
changelog:csrf-protection | changelog | csrf-protection | CSRF protection | Since writable canned queries are built using POST forms, Datasette now ships with CSRF protection ( #798 ). This applies automatically to any POST request, which means plugins need to include a csrftoken in any POST forms that they render. They can do that like so: <input type="hidden" name="csrftoken" value="{{ csrftoken() }}"> | ["Changelog", "0.44 (2020-06-11)"] | [{"href": "https://github.com/simonw/datasette/issues/798", "label": "#798"}] |
deploying:deploying-buildpacks | deploying | deploying-buildpacks | Deploying using buildpacks | Some hosting providers such as Heroku , DigitalOcean App Platform and Scalingo support the Buildpacks standard for deploying Python web applications. Deploying Datasette on these platforms requires two files: requirements.txt and Procfile . The requirements.txt file lets the platform know which Python packages should be installed. It should contain datasette at a minimum, but can also list any Datasette plugins you wish to install - for example: datasette datasette-vega The Procfile lets the hosting platform know how to run the command that serves web traffic. It should look like this: web: datasette . -h 0.0.0.0 -p $PORT --cors The $PORT environment variable is provided by the hosting platform. --cors enables CORS requests from JavaScript running on other websites to your domain - omit this if you don't want to allow CORS. You can add additional Datasette Settings options here too. These two files should be enough to deploy Datasette on any host that supports buildpacks. Datasette will serve any SQLite files that are included in the root directory of the application. If you want to build SQLite files or download them as part of the deployment process you can do so using a bin/post_compile file. For example, the following bin/post_compile will download an example database that will then be served by Datasette: wget https://fivethirtyeight.datasettes.com/fivethirtyeight.db simonw/buildpack-datasette-demo is an example GitHub repository showing a Datasette configuration that can be deployed to a buildpack-supporting host. | ["Deploying Datasette"] | [{"href": "https://www.heroku.com/", "label": "Heroku"}, {"href": "https://www.digitalocean.com/docs/app-platform/", "label": "DigitalOcean App Platform"}, {"href": "https://scalingo.com/", "label": "Scalingo"}, {"href": "https://buildpacks.io/", "label": "Buildpacks standard"}, {"href": "https://github.com/simonw/buildpack-datasette-demo", "label": "simonw/buildpack-datasette-demo"}] |
contributing:contributing-documentation-cog | contributing | contributing-documentation-cog | Running Cog | Some pages of documentation (in particular the CLI reference ) are automatically updated using Cog . To update these pages, run the following command: cog -r docs/*.rst | ["Contributing", "Editing and building the documentation"] | [{"href": "https://github.com/nedbat/cog", "label": "Cog"}] |
sql_queries:fragment | sql_queries | fragment | fragment | Some plugins, such as datasette-vega , can be configured by including additional data in the fragment hash of the URL - the bit that comes after a # symbol. You can set a default fragment hash that will be included in the link to the canned query from the database index page using the "fragment" key. This example demonstrates both fragment and hide_sql : { "databases": { "fixtures": { "queries": { "neighborhood_search": { "sql": "select neighborhood, facet_cities.name, state\nfrom facetable join facet_cities on facetable.city_id = facet_cities.id\nwhere neighborhood like '%' || :text || '%' order by neighborhood;", "fragment": "fragment-goes-here", "hide_sql": true } } } } } See here for a demo of this in action. | ["Running SQL queries", "Canned queries", "Additional canned query options"] | [{"href": "https://github.com/simonw/datasette-vega", "label": "datasette-vega"}, {"href": "https://latest.datasette.io/fixtures#queries", "label": "See here"}] |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [sections] ( [id] TEXT PRIMARY KEY, [page] TEXT, [ref] TEXT, [title] TEXT, [content] TEXT, [breadcrumbs] TEXT, [references] TEXT );