sections
478 rows sorted by content descending
This data as json, CSV (advanced)
id | page | ref | title | content ▲ | breadcrumbs | references |
---|---|---|---|---|---|---|
internals:internals-datasette | internals | internals-datasette | Datasette class | This object is an instance of the Datasette class, passed to many plugin hooks as an argument called datasette . You can create your own instance of this - for example to help write tests for a plugin - like so: from datasette.app import Datasette # With no arguments a single in-memory database will be attached datasette = Datasette() # The files= argument can load files from disk datasette = Datasette(files=["/path/to/my-database.db"]) # Pass metadata as a JSON dictionary like this datasette = Datasette( files=["/path/to/my-database.db"], metadata={ "databases": { "my-database": { "description": "This is my database" } } }, ) Constructor parameters include: files=[...] - a list of database files to open immutables=[...] - a list of database files to open in immutable mode metadata={...} - a dictionary of Metadata config_dir=... - the configuration directory to use, stored in datasette.config_dir | ["Internals for plugins"] | [] |
internals:database-execute-write-fn | internals | database-execute-write-fn | await db.execute_write_fn(fn, block=True) | This method works like .execute_write() , but instead of a SQL statement you give it a callable Python function. Your function will be queued up and then called when the write connection is available, passing that connection as the argument to the function. The function can then perform multiple actions, safe in the knowledge that it has exclusive access to the single writable connection for as long as it is executing. fn needs to be a regular function, not an async def function. For example: def delete_and_return_count(conn): conn.execute("delete from some_table where id > 5") return conn.execute( "select count(*) from some_table" ).fetchone()[0] try: num_rows_left = await database.execute_write_fn( delete_and_return_count ) except Exception as e: print("An error occurred:", e) The value returned from await database.execute_write_fn(...) will be the return value from your function. If your function raises an exception that exception will be propagated up to the await line. If you specify block=False the method becomes fire-and-forget, queueing your function to be executed and then allowing your code after the call to .execute_write_fn() to continue running while the underlying thread waits for an opportunity to run your function. A UUID representing the queued task will be returned. Any exceptions in your code will be silently swallowed. | ["Internals for plugins", "Database class"] | [] |
settings:setting-facet-time-limit-ms | settings | setting-facet-time-limit-ms | facet_time_limit_ms | This is the time limit Datasette allows for calculating a facet, which defaults to 200ms: datasette mydatabase.db --setting facet_time_limit_ms 1000 | ["Settings", "Settings"] | [] |
changelog:id142 | changelog | id142 | 0.19 (2018-04-16) | This is the first preview of the new Datasette plugins mechanism. Only two plugin hooks are available so far - for custom SQL functions and custom template filters. There's plenty more to come - read the documentation and get involved in the tracking ticket if you have feedback on the direction so far. Fix for _sort_desc=sortable_with_nulls test, refs #216 Fixed #216 - paginate correctly when sorting by nullable column Initial documentation for plugins, closes #213 https://docs.datasette.io/en/stable/plugins.html New --plugins-dir=plugins/ option ( #212 ) New option causing Datasette to load and evaluate all of the Python files in the specified directory and register any plugins that are defined in those files. This new option is available for the following commands: datasette serve mydb.db --plugins-dir=plugins/ datasette publish now/heroku mydb.db --plugins-dir=plugins/ datasette package mydb.db --plugins-dir=plugins/ Start of the plugin system, based on pluggy ( #210 ) Uses https://pluggy.readthedocs.io/ originally created for the py.test project We're starting with two plugin hooks: prepare_connection(conn) This is called when a new SQLite connection is created. It can be used to register custom SQL functions. prepare_jinja2_environment(env) This is called with the Jinja2 environment. It can be used to register custom template tags and filters. An example plugin which… | ["Changelog"] | [{"href": "https://docs.datasette.io/en/stable/plugins.html", "label": "the documentation"}, {"href": "https://github.com/simonw/datasette/issues/14", "label": "the tracking ticket"}, {"href": "https://github.com/simonw/datasette/issues/216", "label": "#216"}, {"href": "https://github.com/simonw/datasette/issues/216", "label": "#216"}, {"href": "https://github.com/simonw/datasette/issues/213", "label": "#213"}, {"href": "https://docs.datasette.io/en/stable/plugins.html", "label": "https://docs.datasette.io/en/stable/plugins.html"}, {"href": "https://github.com/simonw/datasette/issues/212", "label": "#212"}, {"href": "https://github.com/simonw/datasette/issues/14", "label": "#210"}, {"href": "https://pluggy.readthedocs.io/", "label": "https://pluggy.readthedocs.io/"}, {"href": "https://github.com/simonw/datasette-plugin-demos", "label": "https://github.com/simonw/datasette-plugin-demos"}, {"href": "https://github.com/simonw/datasette/issues/14", "label": "#14"}] |
plugin_hooks:plugin-hook-startup | plugin_hooks | plugin-hook-startup | startup(datasette) | This hook fires when the Datasette application server first starts up. You can implement a regular function, for example to validate required plugin configuration: @hookimpl def startup(datasette): config = datasette.plugin_config("my-plugin") or {} assert ( "required-setting" in config ), "my-plugin requires setting required-setting" Or you can return an async function which will be awaited on startup. Use this option if you need to make any database queries: @hookimpl def startup(datasette): async def inner(): db = datasette.get_database() if "my_table" not in await db.table_names(): await db.execute_write( """ create table my_table (mycol text) """ ) return inner Potential use-cases: Run some initialization code for the plugin Create database tables that a plugin needs on startup Validate the metadata configuration for a plugin on startup, and raise an error if it is invalid If you are writing unit tests for a plugin that uses this hook and doesn't exercise Datasette by sending any simulated requests through it you will need to explicitly call await ds.invoke_startup() in your tests. An example: @pytest.mark.asyncio async def test_my_plugin(): ds = Datasette() await ds.invoke_startup() # Rest of test goes here Examples: datasette-saved-queries , datasette-init | ["Plugin hooks"] | [{"href": "https://datasette.io/plugins/datasette-saved-queries", "label": "datasette-saved-queries"}, {"href": "https://datasette.io/plugins/datasette-init", "label": "datasette-init"}] |
internals:internals-utils-parse-metadata | internals | internals-utils-parse-metadata | parse_metadata(content) | This function accepts a string containing either JSON or YAML, expected to be of the format described in Metadata . It returns a nested Python dictionary representing the parsed data from that string. If the metadata cannot be parsed as either JSON or YAML the function will raise a utils.BadMetadataError exception. datasette.utils. parse_metadata content : str dict Detects if content is JSON or YAML and parses it appropriately. | ["Internals for plugins", "The datasette.utils module"] | [] |
cli-reference:cli-help-serve-help | cli-reference | cli-help-serve-help | datasette serve | This command starts the Datasette web application running on your machine: datasette serve mydatabase.db Or since this is the default command you can run this instead: datasette mydatabase.db Once started you can access it at http://localhost:8001 [[[cog help(["serve", "--help"]) ]]] Usage: datasette serve [OPTIONS] [FILES]... Serve up specified SQLite database files with a web UI Options: -i, --immutable PATH Database files to open in immutable mode -h, --host TEXT Host for server. Defaults to 127.0.0.1 which means only connections from the local machine will be allowed. Use 0.0.0.0 to listen to all IPs and allow access from other machines. -p, --port INTEGER RANGE Port for server, defaults to 8001. Use -p 0 to automatically assign an available port. [0<=x<=65535] --uds TEXT Bind to a Unix domain socket --reload Automatically reload if code or metadata change detected - useful for development --cors Enable CORS by serving Access-Control-Allow- Origin: * --load-extension PATH:ENTRYPOINT? Path to a SQLite extension to load, and optional entrypoint --inspect-file TEXT Path to JSON file created using "datasette inspect" -m, --metadata FILENAME Path to JSON/YAML file containing license/source metadata --template-dir DIRECTORY Path to directory containing custom templates --plugins-dir DIRECTORY Path to directory containing custom plugins --static MOUNT:DIRECTORY Serve static files fr… | ["CLI reference"] | [] |
cli-reference:cli-help-serve-help-settings | cli-reference | cli-help-serve-help-settings | datasette serve --help-settings | This command outputs all of the available Datasette settings . These can be passed to datasette serve using datasette serve --setting name value . [[[cog help(["--help-settings"]) ]]] Settings: default_page_size Default page size for the table view (default=100) max_returned_rows Maximum rows that can be returned from a table or custom query (default=1000) num_sql_threads Number of threads in the thread pool for executing SQLite queries (default=3) sql_time_limit_ms Time limit for a SQL query in milliseconds (default=1000) default_facet_size Number of values to return for requested facets (default=30) facet_time_limit_ms Time limit for calculating a requested facet (default=200) facet_suggest_time_limit_ms Time limit for calculating a suggested facet (default=50) allow_facet Allow users to specify columns to facet using ?_facet= parameter (default=True) default_allow_sql Allow anyone to run arbitrary SQL queries (default=True) allow_download Allow users to download the original SQLite database files (default=True) suggest_facets Calculate and display suggested facets (default=True) default_cache_ttl Default HTTP cache TTL (used in Cache-Control: max-age= header) (default=5) cache_size_kb SQLite cache size in KB (0 == use SQLite default) (default=0) allow_csv_stream Allow .csv?_stream=1 to download all rows (ignoring max_returned… | ["CLI reference", "datasette serve"] | [] |
internals:internals-internal | internals | internals-internal | The _internal database | This API should be considered unstable - the structure of these tables may change prior to the release of Datasette 1.0. Datasette maintains an in-memory SQLite database with details of the the databases, tables and columns for all of the attached databases. By default all actors are denied access to the view-database permission for the _internal database, so the database is not visible to anyone unless they sign in as root . Plugins can access this database by calling db = datasette.get_database("_internal") and then executing queries using the Database API . You can explore an example of this database by signing in as root to the latest.datasette.io demo instance and then navigating to latest.datasette.io/_internal . | ["Internals for plugins"] | [{"href": "https://latest.datasette.io/login-as-root", "label": "signing in as root"}, {"href": "https://latest.datasette.io/_internal", "label": "latest.datasette.io/_internal"}] |
full_text_search:full-text-search-fts-versions | full_text_search | full-text-search-fts-versions | FTS versions | There are three different versions of the SQLite FTS module: FTS3, FTS4 and FTS5. You can tell which versions are supported by your instance of Datasette by checking the /-/versions page. FTS5 is the most advanced module but may not be available in the SQLite version that is bundled with your Python installation. Most importantly, FTS5 is the only version that has the ability to order by search relevance without needing extra code. If you can't be sure that FTS5 will be available, you should use FTS4. | ["Full-text search"] | [] |
changelog:id34 | changelog | id34 | 0.54 (2021-01-25) | The two big new features in this release are the _internal SQLite in-memory database storing details of all connected databases and tables, and support for JavaScript modules in plugins and additional scripts. For additional commentary on this release, see Datasette 0.54, the annotated release notes . | ["Changelog"] | [{"href": "https://simonwillison.net/2021/Jan/25/datasette/", "label": "Datasette 0.54, the annotated release notes"}] |
metadata:metadata-source-license-about | metadata | metadata-source-license-about | Source, license and about | The three visible metadata fields you can apply to everything, specific databases or specific tables are source, license and about. All three are optional. source and source_url should be used to indicate where the underlying data came from. license and license_url should be used to indicate the license under which the data can be used. about and about_url can be used to link to further information about the project - an accompanying blog entry for example. For each of these you can provide just the *_url field and Datasette will treat that as the default link label text and display the URL directly on the page. | ["Metadata"] | [] |
changelog:id171 | changelog | id171 | 0.14 (2017-12-09) | The theme of this release is customization: Datasette now allows every aspect of its presentation to be customized either using additional CSS or by providing entirely new templates. Datasette's metadata.json format has also been expanded, to allow per-database and per-table metadata. A new datasette skeleton command can be used to generate a skeleton JSON file ready to be filled in with per-database and per-table details. The metadata.json file can also be used to define canned queries , as a more powerful alternative to SQL views. extra_css_urls / extra_js_urls in metadata A mechanism in the metadata.json format for adding custom CSS and JS urls. Create a metadata.json file that looks like this: { "extra_css_urls": [ "https://simonwillison.net/static/css/all.bf8cd891642c.css" ], "extra_js_urls": [ "https://code.jquery.com/jquery-3.2.1.slim.min.js" ] } Then start datasette like this: datasette mydb.db --metadata=metadata.json The CSS and JavaScript files will be linked in the <head> of every page. You can also specify a SRI (subresource integrity hash) for these assets: { "extra_css_urls": [ { "url": "https://simonwillison.net/static/css/all.bf8cd891642c.css", "sri": "sha384-9qIZekWUyjCyDIf2YK1FRoKiPJq4PHt6tp/ulnuuyRBvazd0hG7pWbE99zvwSznI" } ], "extra_js_urls": [ { "url": "https://code.jquery.com/jquery-3.2.1.slim.min.js", "sri": "sha256-k2WSCIexGzOj3Euiig+TlR8gA0EmPjuc79OEeY5L45g=" } ] } Modern browsers will only execute the stylesheet or JavaScript if the SRI hash … | ["Changelog"] | [{"href": "https://docs.datasette.io/en/stable/custom_templates.html", "label": "to be customized"}, {"href": "https://docs.datasette.io/en/stable/metadata.html", "label": "metadata.json format"}, {"href": "https://docs.datasette.io/en/stable/sql_queries.html#canned-queries", "label": "canned queries"}, {"href": "https://www.srihash.org/", "label": "https://www.srihash.org/"}, {"href": "https://github.com/simonw/datasette/issues/153", "label": "#153"}, {"href": "https://github.com/simonw/datasette/issues/153", "label": "#153"}, {"href": "https://github.com/simonw/datasette/issues/160", "label": "#160"}, {"href": "https://github.com/simonw/datasette/issues/164", "label": "#164"}, {"href": "https://github.com/simonw/datasette/issues/165", "label": "#165"}, {"href": "https://github.com/simonw/datasette/issues/130", "label": "#130"}, {"href": "https://github.com/simonw/datasette/issues/168", "label": "#168"}, {"href": "https://github.com/channelcat/sanic/releases/tag/0.7.0", "label": "https://github.com/channelcat/sanic/releases/tag/0.7.0"}, {"href": "https://github.com/simonw/datasette/issues/171", "label": "#171"}] |
pages:tableview | pages | tableview | Table | The table page is the heart of Datasette: it allows users to interactively explore the contents of a database table, including sorting, filtering, Full-text search and applying Facets . The HTML interface is worth spending some time exploring. As with other pages, you can return the JSON data by appending .json to the URL path, before any ? query string arguments. The query string arguments are described in more detail here: Table arguments You can also use the table page to interactively construct a SQL query - by applying different filters and a sort order for example - and then click the "View and edit SQL" link to see the SQL query that was used for the page and edit and re-submit it. Some examples: ../items lists all of the line-items registered by UK MPs as potential conflicts of interest. It demonstrates Datasette's support for Full-text search . ../antiquities-act%2Factions_under_antiquities_act is an interface for exploring the "actions under the antiquities act" data table published by FiveThirtyEight. ../global-power-plants?country_long=United+Kingdom&primary_fuel=Gas is a filtered table page showing every Gas power plant in the United Kingdom. It includes some default facets (configured using its metadata.json ) and uses the datasette-cluster-map plugin to show a map of the results. | ["Pages and API endpoints"] | [{"href": "https://register-of-members-interests.datasettes.com/regmem/items", "label": "../items"}, {"href": "https://fivethirtyeight.datasettes.com/fivethirtyeight/antiquities-act%2Factions_under_antiquities_act", "label": "../antiquities-act%2Factions_under_antiquities_act"}, {"href": "https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=owner&_facet=country_long&country_long__exact=United+Kingdom&primary_fuel=Gas", "label": "../global-power-plants?country_long=United+Kingdom&primary_fuel=Gas"}, {"href": "https://global-power-plants.datasettes.com/-/metadata", "label": "its metadata.json"}, {"href": "https://github.com/simonw/datasette-cluster-map", "label": "datasette-cluster-map"}] |
authentication:authentication-permissions-allow | authentication | authentication-permissions-allow | Defining permissions with "allow" blocks | The standard way to define permissions in Datasette is to use an "allow" block. This is a JSON document describing which actors are allowed to perform a permission. The most basic form of allow block is this ( allow demo , deny demo ): { "allow": { "id": "root" } } This will match any actors with an "id" property of "root" - for example, an actor that looks like this: { "id": "root", "name": "Root User" } An allow block can specify "deny all" using false ( demo ): { "allow": false } An "allow" of true allows all access ( demo ): { "allow": true } Allow keys can provide a list of values. These will match any actor that has any of those values ( allow demo , deny demo ): { "allow": { "id": ["simon", "cleopaws"] } } This will match any actor with an "id" of either "simon" or "cleopaws" . Actors can have properties that feature a list of values. These will be matched against the list of values in an allow block. Consider the following actor: { "id": "simon", "roles": ["staff", "developer"] } This allow block will provide access to any actor that has "developer" as one of their roles ( allow demo , deny demo ): { "allow": { "roles": ["developer"] } } Note that "roles" is not a concept that is baked into Datasette - it's a convention that plugins can choose to implement and act on. If you want to provide access to any actor with a value for a specific key, use "*" . For example, to match any logged-in user specify the following ( allow demo , deny demo ): { "allow": { "id": "*" } } You can specify that only unauthenticated actors (from anynomous HTTP requests) should be all… | ["Authentication and permissions", "Permissions"] | [{"href": "https://latest.datasette.io/-/allow-debug?actor=%7B%22id%22%3A+%22root%22%7D&allow=%7B%0D%0A++++++++%22id%22%3A+%22root%22%0D%0A++++%7D", "label": "allow demo"}, {"href": "https://latest.datasette.io/-/allow-debug?actor=%7B%22id%22%3A+%22trevor%22%7D&allow=%7B%0D%0A++++++++%22id%22%3A+%22root%22%0D%0A++++%7D", "label": "deny demo"}, {"href": "https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22id%22%3A+%22root%22%0D%0A%7D&allow=false", "label": "demo"}, {"href": "https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22id%22%3A+%22root%22%0D%0A%7D&allow=true", "label": "demo"}, {"href": "https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22id%22%3A+%22cleopaws%22%0D%0A%7D&allow=%7B%0D%0A++++%22id%22%3A+%5B%0D%0A++++++++%22simon%22%2C%0D%0A++++++++%22cleopaws%22%0D%0A++++%5D%0D%0A%7D", "label": "allow demo"}, {"href": "https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22id%22%3A+%22pancakes%22%0D%0A%7D&allow=%7B%0D%0A++++%22id%22%3A+%5B%0D%0A++++++++%22simon%22%2C%0D%0A++++++++%22cleopaws%22%0D%0A++++%5D%0D%0A%7D", "label": "deny demo"}, {"href": "https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22id%22%3A+%22simon%22%2C%0D%0A++++%22roles%22%3A+%5B%0D%0A++++++++%22staff%22%2C%0D%0A++++++++%22developer%22%0D%0A++++%5D%0D%0A%7D&allow=%7B%0D%0A++++%22roles%22%3A+%5B%0D%0A++++++++%22developer%22%0D%0A++++%5D%0D%0A%7D", "label": "allow demo"}, {"href": "https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22id%22%3A+%22cleopaws%22%2C%0D%0A++++%22roles%22%3A+%5B%22dog%22%5D%0D%0A%7D&allow=%7B%0D%0A++++%22roles%22%3A+%5B%0D%0A++++++++%22developer%22%0D%0A++++%5D%0D%0A%7D", "label": "deny demo"}, {"href": "https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22id%22%3A+%22simon%22%0D%0A%7D&allow=%7B%0D%0A++++%22id%22%3A+%22*%22%0D%0A%7D", "label": "allow demo"}, {"href": "https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22bot%22%3A+%22readme-bot%22%0D%0A%7D&allow=%7B%0D%0A++++%22id%22%3A+%22*%22%0D%0A%7D", "label": "deny demo"… |
pages:indexview | pages | indexview | Top-level index | The root page of any Datasette installation is an index page that lists all of the currently attached databases. Some examples: fivethirtyeight.datasettes.com global-power-plants.datasettes.com register-of-members-interests.datasettes.com Add /.json to the end of the URL for the JSON version of the underlying data: fivethirtyeight.datasettes.com/.json global-power-plants.datasettes.com/.json register-of-members-interests.datasettes.com/.json | ["Pages and API endpoints"] | [{"href": "https://fivethirtyeight.datasettes.com/", "label": "fivethirtyeight.datasettes.com"}, {"href": "https://global-power-plants.datasettes.com/", "label": "global-power-plants.datasettes.com"}, {"href": "https://register-of-members-interests.datasettes.com/", "label": "register-of-members-interests.datasettes.com"}, {"href": "https://fivethirtyeight.datasettes.com/.json", "label": "fivethirtyeight.datasettes.com/.json"}, {"href": "https://global-power-plants.datasettes.com/.json", "label": "global-power-plants.datasettes.com/.json"}, {"href": "https://register-of-members-interests.datasettes.com/.json", "label": "register-of-members-interests.datasettes.com/.json"}] |
internals:internals-request | internals | internals-request | Request object | The request object is passed to various plugin hooks. It represents an incoming HTTP request. It has the following properties: .scope - dictionary The ASGI scope that was used to construct this request, described in the ASGI HTTP connection scope specification. .method - string The HTTP method for this request, usually GET or POST . .url - string The full URL for this request, e.g. https://latest.datasette.io/fixtures . .scheme - string The request scheme - usually https or http . .headers - dictionary (str -> str) A dictionary of incoming HTTP request headers. Header names have been converted to lowercase. .cookies - dictionary (str -> str) A dictionary of incoming cookies .host - string The host header from the incoming request, e.g. latest.datasette.io or localhost . .path - string The path of the request excluding the query string, e.g. /fixtures . .full_path - string The path of the… | ["Internals for plugins"] | [{"href": "https://asgi.readthedocs.io/en/latest/specs/www.html#connection-scope", "label": "ASGI HTTP connection scope"}] |
writing_plugins:writing-plugins-one-off | writing_plugins | writing-plugins-one-off | Writing one-off plugins | The quickest way to start writing a plugin is to create a my_plugin.py file and drop it into your plugins/ directory. Here is an example plugin, which adds a new custom SQL function called hello_world() which takes no arguments and returns the string Hello world! . from datasette import hookimpl @hookimpl def prepare_connection(conn): conn.create_function( "hello_world", 0, lambda: "Hello world!" ) If you save this in plugins/my_plugin.py you can then start Datasette like this: datasette serve mydb.db --plugins-dir=plugins/ Now you can navigate to http://localhost:8001/mydb and run this SQL: select hello_world(); To see the output of your plugin. | ["Writing plugins"] | [{"href": "http://localhost:8001/mydb", "label": "http://localhost:8001/mydb"}] |
deploying:deploying | deploying | deploying | Deploying Datasette | The quickest way to deploy a Datasette instance on the internet is to use the datasette publish command, described in Publishing data . This can be used to quickly deploy Datasette to a number of hosting providers including Heroku, Google Cloud Run and Vercel. You can deploy Datasette to other hosting providers using the instructions on this page. | [] | [] |
changelog:better-plugin-documentation | changelog | better-plugin-documentation | Better plugin documentation | The plugin documentation has been re-arranged into four sections, including a brand new section on testing plugins. ( #687 ) Plugins introduces Datasette's plugin system and describes how to install and configure plugins. Writing plugins describes how to author plugins, from one-off single file plugins to packaged plugins that can be published to PyPI. It also describes how to start a plugin using the new datasette-plugin cookiecutter template. Plugin hooks is a full list of detailed documentation for every Datasette plugin hook. Testing plugins describes how to write tests for Datasette plugins, using pytest and HTTPX . | ["Changelog", "0.45 (2020-07-01)"] | [{"href": "https://github.com/simonw/datasette/issues/687", "label": "#687"}, {"href": "https://github.com/simonw/datasette-plugin", "label": "datasette-plugin"}, {"href": "https://docs.pytest.org/", "label": "pytest"}, {"href": "https://www.python-httpx.org/", "label": "HTTPX"}] |
facets:speeding-up-facets-with-indexes | facets | speeding-up-facets-with-indexes | Speeding up facets with indexes | The performance of facets can be greatly improved by adding indexes on the columns you wish to facet by. Adding indexes can be performed using the sqlite3 command-line utility. Here's how to add an index on the state column in a table called Food_Trucks : $ sqlite3 mydatabase.db SQLite version 3.19.3 2017-06-27 16:48:08 Enter ".help" for usage hints. sqlite> CREATE INDEX Food_Trucks_state ON Food_Trucks("state"); Or using the sqlite-utils command-line utility: $ sqlite-utils create-index mydatabase.db Food_Trucks state | ["Facets"] | [{"href": "https://sqlite-utils.datasette.io/en/stable/cli.html#creating-indexes", "label": "sqlite-utils"}] |
authentication:logoutview | authentication | logoutview | The /-/logout page | The page at /-/logout provides the ability to log out of a ds_actor cookie authentication session. | ["Authentication and permissions", "The ds_actor cookie"] | [] |
changelog:faceting | changelog | faceting | Faceting | The number of unique values in a facet is now always displayed. Previously it was only displayed if the user specified ?_facet_size=max . ( #1556 ) Facets of type date or array can now be configured in metadata.json , see Facets in metadata.json . Thanks, David Larlet. ( #1552 ) New ?_nosuggest=1 parameter for table views, which disables facet suggestion. ( #1557 ) Fixed bug where ?_facet_array=tags&_facet=tags would only display one of the two selected facets. ( #625 ) | ["Changelog", "0.60 (2022-01-13)"] | [{"href": "https://github.com/simonw/datasette/issues/1556", "label": "#1556"}, {"href": "https://github.com/simonw/datasette/issues/1552", "label": "#1552"}, {"href": "https://github.com/simonw/datasette/issues/1557", "label": "#1557"}, {"href": "https://github.com/simonw/datasette/issues/625", "label": "#625"}] |
changelog:url-building | changelog | url-building | URL building | The new datasette.urls family of methods can be used to generate URLs to key pages within the Datasette interface, both within custom templates and Datasette plugins. See Building URLs within plugins for more details. ( #904 ) | ["Changelog", "0.51 (2020-10-31)"] | [{"href": "https://github.com/simonw/datasette/issues/904", "label": "#904"}] |
changelog:through-for-joins-through-many-to-many-tables | changelog | through-for-joins-through-many-to-many-tables | ?_through= for joins through many-to-many tables | The new ?_through={json} argument to the Table view allows records to be filtered based on a many-to-many relationship. See Special table arguments for full documentation - here's an example . ( #355 ) This feature was added to help support facet by many-to-many , which isn't quite ready yet but will be coming in the next Datasette release. | ["Changelog", "0.29 (2019-07-07)"] | [{"href": "https://latest.datasette.io/fixtures/roadside_attractions?_through={%22table%22:%22roadside_attraction_characteristics%22,%22column%22:%22characteristic_id%22,%22value%22:%221%22}", "label": "an example"}, {"href": "https://github.com/simonw/datasette/issues/355", "label": "#355"}, {"href": "https://github.com/simonw/datasette/issues/551", "label": "facet by many-to-many"}] |
settings:setting-max-csv-mb | settings | setting-max-csv-mb | max_csv_mb | The maximum size of CSV that can be exported, in megabytes. Defaults to 100MB. You can disable the limit entirely by settings this to 0: datasette mydatabase.db --setting max_csv_mb 0 | ["Settings", "Settings"] | [] |
changelog:id59 | changelog | id59 | 0.43 (2020-05-28) | The main focus of this release is a major upgrade to the register_output_renderer(datasette) plugin hook, which allows plugins to provide new output formats for Datasette such as datasette-atom and datasette-ics . Redesign of register_output_renderer(datasette) to provide more context to the render callback and support an optional "can_render" callback that controls if a suggested link to the output format is provided. ( #581 , #770 ) Visually distinguish float and integer columns - useful for figuring out why order-by-column might be returning unexpected results. ( #729 ) The Request object , which is passed to several plugin hooks, is now documented. ( #706 ) New metadata.json option for setting a custom default page size for specific tables and views, see Setting a custom page size . ( #751 ) Canned queries can now be configured with a default URL fragment hash, useful when working with plugins such as datasette-vega , see Additional canned query options . ( #706 ) Fixed a bug in datasette publish when running on operating systems where the /tmp directory lives in a different volume, using a backport of the Python 3.8 shutil.copytree() function. ( #744 ) Every plugin hook is now covered by the unit tests, and a new unit test checks that each plugin hook has at least one corresponding test. ( #771 , #773 ) | ["Changelog"] | [{"href": "https://github.com/simonw/datasette-atom", "label": "datasette-atom"}, {"href": "https://github.com/simonw/datasette-ics", "label": "datasette-ics"}, {"href": "https://github.com/simonw/datasette/issues/581", "label": "#581"}, {"href": "https://github.com/simonw/datasette/issues/770", "label": "#770"}, {"href": "https://github.com/simonw/datasette/issues/729", "label": "#729"}, {"href": "https://github.com/simonw/datasette/issues/706", "label": "#706"}, {"href": "https://github.com/simonw/datasette/issues/751", "label": "#751"}, {"href": "https://github.com/simonw/datasette-vega", "label": "datasette-vega"}, {"href": "https://github.com/simonw/datasette/issues/706", "label": "#706"}, {"href": "https://github.com/simonw/datasette/issues/744", "label": "#744"}, {"href": "https://github.com/simonw/datasette/issues/771", "label": "#771"}, {"href": "https://github.com/simonw/datasette/issues/773", "label": "#773"}] |
changelog:id46 | changelog | id46 | 0.50 (2020-10-09) | The key new feature in this release is the column actions menu on the table page ( #891 ). This can be used to sort a column in ascending or descending order, facet data by that column or filter the table to just rows that have a value for that column. Plugin authors can use the new datasette.client object to make internal HTTP requests from their plugins, allowing them to make use of Datasette's JSON API. ( #943 ) New Deploying Datasette documentation with guides for deploying Datasette on a Linux server using systemd or to hosting providers that support buildpacks . ( #514 , #997 ) Other improvements in this release: Publishing to Google Cloud Run documentation now covers Google Cloud SDK options. Thanks, Geoffrey Hing. ( #995 ) New datasette -o option which opens your browser as soon as Datasette starts up. ( #970 ) Datasette now sets sqlite3.enable_callback_tracebacks(True) so that errors in custom SQL functions will display tracebacks. ( #891 ) Fixed two rendering bugs with column headers in portrait mobile view. ( #978 , #980 ) New db.table_column_details(table) introspection method for retrieving full details of the columns in a specific table, see Database introspection . Fixed a routing bug with custom page wildcard templates. ( #996 ) datasette publish heroku now deploys using Python 3.8.6. New datasette publish heroku --tar= option. ( #969 ) OPTIONS requests against HTML pages no longer return a 500 error. ( #1001 ) … | ["Changelog"] | [{"href": "https://github.com/simonw/datasette/issues/891", "label": "#891"}, {"href": "https://github.com/simonw/datasette/issues/943", "label": "#943"}, {"href": "https://github.com/simonw/datasette/issues/514", "label": "#514"}, {"href": "https://github.com/simonw/datasette/issues/997", "label": "#997"}, {"href": "https://github.com/simonw/datasette/pull/995", "label": "#995"}, {"href": "https://github.com/simonw/datasette/issues/970", "label": "#970"}, {"href": "https://github.com/simonw/datasette/issues/891", "label": "#891"}, {"href": "https://github.com/simonw/datasette/issues/978", "label": "#978"}, {"href": "https://github.com/simonw/datasette/issues/980", "label": "#980"}, {"href": "https://github.com/simonw/datasette/issues/996", "label": "#996"}, {"href": "https://github.com/simonw/datasette/issues/969", "label": "#969"}, {"href": "https://github.com/simonw/datasette/issues/1001", "label": "#1001"}, {"href": "https://simonwillison.net/2020/Oct/9/datasette-0-50/", "label": "Datasette 0.50: The annotated release notes"}] |
csv_export:csv-export-url-parameters | csv_export | csv-export-url-parameters | URL parameters | The following options can be used to customize the CSVs returned by Datasette. ?_header=off This removes the first row of the CSV file specifying the headings - only the row data will be returned. ?_stream=on Stream all matching records, not just the first page of results. See below. ?_dl=on Causes Datasette to return a content-disposition: attachment; filename="filename.csv" header. | ["CSV export"] | [] |
settings:id2 | settings | id2 | Settings | The following options can be set using --setting name value , or by storing them in the settings.json file for use with Configuration directory mode . | ["Settings"] | [] |
internals:internals-shortcuts | internals | internals-shortcuts | Import shortcuts | The following commonly used symbols can be imported directly from the datasette module: from datasette import Response from datasette import Forbidden from datasette import NotFound from datasette import hookimpl from datasette import actor_matches_allow | ["Internals for plugins"] | [] |
spatialite:installing-spatialite-on-os-x | spatialite | installing-spatialite-on-os-x | Installing SpatiaLite on OS X | The easiest way to install SpatiaLite on OS X is to use Homebrew . brew update brew install spatialite-tools This will install the spatialite command-line tool and the mod_spatialite dynamic library. You can now run Datasette like so: datasette --load-extension=spatialite | ["SpatiaLite", "Installation"] | [{"href": "https://brew.sh/", "label": "Homebrew"}] |
contributing:contributing-continuous-deployment | contributing | contributing-continuous-deployment | Continuously deployed demo instances | The demo instance at latest.datasette.io is re-deployed automatically to Google Cloud Run for every push to main that passes the test suite. This is implemented by the GitHub Actions workflow at .github/workflows/deploy-latest.yml . Specific branches can also be set to automatically deploy by adding them to the on: push: branches block at the top of the workflow YAML file. Branches configured in this way will be deployed to a new Cloud Run service whether or not their tests pass. The Cloud Run URL for a branch demo can be found in the GitHub Actions logs. | ["Contributing"] | [{"href": "https://latest.datasette.io/", "label": "latest.datasette.io"}, {"href": "https://github.com/simonw/datasette/blob/main/.github/workflows/deploy-latest.yml", "label": ".github/workflows/deploy-latest.yml"}] |
settings:setting-default-facet-size | settings | setting-default-facet-size | default_facet_size | The default number of unique rows returned by Facets is 30. You can customize it like this: datasette mydatabase.db --setting default_facet_size 50 | ["Settings", "Settings"] | [] |
settings:setting-default-page-size | settings | setting-default-page-size | default_page_size | The default number of rows returned by the table page. You can over-ride this on a per-page basis using the ?_size=80 query string parameter, provided you do not specify a value higher than the max_returned_rows setting. You can set this default using --setting like so: datasette mydatabase.db --setting default_page_size 50 | ["Settings", "Settings"] | [] |
json_api:json-api-shapes | json_api | json-api-shapes | Different shapes | The default JSON representation of data from a SQLite table or custom query looks like this: { "database": "sf-trees", "table": "qSpecies", "columns": [ "id", "value" ], "rows": [ [ 1, "Myoporum laetum :: Myoporum" ], [ 2, "Metrosideros excelsa :: New Zealand Xmas Tree" ], [ 3, "Pinus radiata :: Monterey Pine" ] ], "truncated": false, "next": "100", "next_url": "http://127.0.0.1:8001/sf-trees-02c8ef1/qSpecies.json?_next=100", "query_ms": 1.9571781158447266 } The columns key lists the columns that are being returned, and the rows key then returns a list of lists, each one representing a row. The order of the values in each row corresponds to the columns. The _shape parameter can be used to access alternative formats for the rows key which may be more convenient for your application. There are three options: ?_shape=arrays - "rows" is the default option, shown above ?_shape=objects - "rows" is a list of JSON key/value objects ?_shape=array - an JSON array of objects ?_shape=array&_nl=on - a newline-separated list of JSON objects ?_shape=arrayfirst - a flat JSON array containing just the first value from each row ?_shape=object - a JSON object keyed using the primary keys of the rows _shape=objects looks like this: { "database": "sf-trees", ... "rows": [ { "id": 1, … | ["JSON API"] | [] |
json_api:json-api-pagination | json_api | json-api-pagination | Pagination | The default JSON representation includes a "next_url" key which can be used to access the next page of results. If that key is null or missing then it means you have reached the final page of results. Other representations include pagination information in the link HTTP header. That header will look something like this: link: <https://latest.datasette.io/fixtures/sortable.json?_next=d%2Cv>; rel="next" Here is an example Python function built using requests that returns a list of all of the paginated items from one of these API endpoints: def paginate(url): items = [] while url: response = requests.get(url) try: url = response.links.get("next").get("url") except AttributeError: url = None items.extend(response.json()) return items | ["JSON API"] | [{"href": "https://requests.readthedocs.io/", "label": "requests"}] |
authentication:permissionsdebugview | authentication | permissionsdebugview | The permissions debug tool | The debug tool at /-/permissions is only available to the authenticated root user (or any actor granted the permissions-debug action according to a plugin). It shows the thirty most recent permission checks that have been carried out by the Datasette instance. This is designed to help administrators and plugin authors understand exactly how permission checks are being carried out, in order to effectively configure Datasette's permission system. | ["Authentication and permissions"] | [] |
introspection:messagesdebugview | introspection | messagesdebugview | /-/messages | The debug tool at /-/messages can be used to set flash messages to try out that feature. See .add_message(request, message, type=datasette.INFO) for details of this feature. | ["Introspection"] | [] |
changelog:id158 | changelog | id158 | 0.15 (2018-04-09) | The biggest new feature in this release is the ability to sort by column. On the table page the column headers can now be clicked to apply sort (or descending sort), or you can specify ?_sort=column or ?_sort_desc=column directly in the URL. table_rows => table_rows_count , filtered_table_rows => filtered_table_rows_count Renamed properties. Closes #194 New sortable_columns option in metadata.json to control sort options. You can now explicitly set which columns in a table can be used for sorting using the _sort and _sort_desc arguments using metadata.json : { "databases": { "database1": { "tables": { "example_table": { "sortable_columns": [ "height", "weight" ] } } } } } Refs #189 Column headers now link to sort/desc sort - refs #189 _sort and _sort_desc parameters for table views Allows for paginated sorted results based on a specified column. Refs #189 Total row count now correct even if _next applied Use .custom_sql() for _group_count implementation (refs #150 ) Make HTML title more readable in query template ( #180 ) [Ryan Pitts] New ?_shape=objects/object/lists param for JSON API ( #192 ) New _shape= parameter repl… | ["Changelog"] | [{"href": "https://github.com/simonw/datasette/issues/194", "label": "#194"}, {"href": "https://github.com/simonw/datasette/issues/189", "label": "#189"}, {"href": "https://github.com/simonw/datasette/issues/189", "label": "#189"}, {"href": "https://github.com/simonw/datasette/issues/189", "label": "#189"}, {"href": "https://github.com/simonw/datasette/issues/150", "label": "#150"}, {"href": "https://github.com/simonw/datasette/issues/180", "label": "#180"}, {"href": "https://github.com/simonw/datasette/issues/192", "label": "#192"}, {"href": "https://github.com/simonw/datasette/issues/122", "label": "#122"}, {"href": "https://github.com/simonw/datasette/issues/190", "label": "#190"}, {"href": "https://github.com/simonw/datasette/issues/190", "label": "#190"}, {"href": "https://github.com/simonw/datasette/issues/185", "label": "#185"}, {"href": "https://github.com/simonw/datasette/issues/178", "label": "#178"}] |
changelog:id115 | changelog | id115 | 0.22 (2018-05-20) | The big new feature in this release is Facets . Datasette can now apply faceted browse to any column in any table. It will also suggest possible facets. See the Datasette Facets announcement post for more details. In addition to the work on facets: Added docs for introspection endpoints New --config option, added --help-config , closes #274 Removed the --page_size= argument to datasette serve in favour of: datasette serve --config default_page_size:50 mydb.db Added new help section: $ datasette --help-config Config options: default_page_size Default page size for the table view (default=100) max_returned_rows Maximum rows that can be returned from a table or custom query (default=1000) sql_time_limit_ms Time limit for a SQL query in milliseconds (default=1000) default_facet_size Number of values to return for requested facets (default=30) facet_time_limit_ms Time limit for calculating a requested facet (default=200) facet_suggest_time_limit_ms Time limit for calculating a suggested facet (default=50) Only apply responsive table styles to .rows-and-column Otherwise they interfere with tables in the description, e.g. on https://fivethirtyeight.datasettes.com/fivethirtyeight/nba-elo%2Fnbaallelo Refactored views into new views/ modules, refs #256 Documentation for SQLite full-text search support, closes #253 … | ["Changelog"] | [{"href": "https://simonwillison.net/2018/May/20/datasette-facets/", "label": "Datasette Facets"}, {"href": "https://docs.datasette.io/en/stable/introspection.html", "label": "docs for introspection endpoints"}, {"href": "https://github.com/simonw/datasette/issues/274", "label": "#274"}, {"href": "https://fivethirtyeight.datasettes.com/fivethirtyeight/nba-elo%2Fnbaallelo", "label": "https://fivethirtyeight.datasettes.com/fivethirtyeight/nba-elo%2Fnbaallelo"}, {"href": "https://github.com/simonw/datasette/issues/256", "label": "#256"}, {"href": "https://docs.datasette.io/en/stable/full_text_search.html", "label": "Documentation for SQLite full-text search"}, {"href": "https://github.com/simonw/datasette/issues/253", "label": "#253"}, {"href": "https://github.com/simonw/datasette/issues/252", "label": "#252"}] |
getting_started:getting-started-demo | getting_started | getting-started-demo | Play with a live demo | The best way to experience Datasette for the first time is with a demo: global-power-plants.datasettes.com provides a searchable database of power plants around the world, using data from the World Resources Institude rendered using the datasette-cluster-map plugin. fivethirtyeight.datasettes.com shows Datasette running against over 400 datasets imported from the FiveThirtyEight GitHub repository . | ["Getting started"] | [{"href": "https://global-power-plants.datasettes.com/global-power-plants/global-power-plants", "label": "global-power-plants.datasettes.com"}, {"href": "https://www.wri.org/publication/global-power-plant-database", "label": "World Resources Institude"}, {"href": "https://github.com/simonw/datasette-cluster-map", "label": "datasette-cluster-map"}, {"href": "https://fivethirtyeight.datasettes.com/fivethirtyeight", "label": "fivethirtyeight.datasettes.com"}, {"href": "https://github.com/fivethirtyeight/data", "label": "FiveThirtyEight GitHub repository"}] |
testing_plugins:testing-plugins-datasette-test-instance | testing_plugins | testing-plugins-datasette-test-instance | Setting up a Datasette test instance | The above example shows the easiest way to start writing tests against a Datasette instance: from datasette.app import Datasette import pytest @pytest.mark.asyncio async def test_plugin_is_installed(): datasette = Datasette(memory=True) response = await datasette.client.get("/-/plugins.json") assert response.status_code == 200 Creating a Datasette() instance like this as useful shortcut in tests, but there is one detail you need to be aware of. It's important to ensure that the async method .invoke_startup() is called on that instance. You can do that like this: datasette = Datasette(memory=True) await datasette.invoke_startup() This method registers any startup(datasette) or prepare_jinja2_environment(env, datasette) plugins that might themselves need to make async calls. If you are using await datasette.client.get() and similar methods then you don't need to worry about this - Datasette automatically calls invoke_startup() the first time it handles a request. | ["Testing plugins"] | [] |
spatialite:spatialite-warning | spatialite | spatialite-warning | Warning | The SpatiaLite extension adds a large number of additional SQL functions , some of which are not be safe for untrusted users to execute: they may cause the Datasette server to crash. You should not expose a SpatiaLite-enabled Datasette instance to the public internet without taking extra measures to secure it against potentially harmful SQL queries. The following steps are recommended: Disable arbitrary SQL queries by untrusted users. See Controlling the ability to execute arbitrary SQL for ways to do this. The easiest is to start Datasette with the datasette --setting default_allow_sql off option. Define Canned queries with the SQL queries that use SpatiaLite functions that you want people to be able to execute. The Datasette SpatiaLite tutorial includes detailed instructions for running SpatiaLite safely using these techniques | ["SpatiaLite"] | [{"href": "https://www.gaia-gis.it/gaia-sins/spatialite-sql-5.0.1.html", "label": "a large number of additional SQL functions"}, {"href": "https://datasette.io/tutorials/spatialite", "label": "Datasette SpatiaLite tutorial"}] |
pages:pages | pages | pages | Pages and API endpoints | The Datasette web application offers a number of different pages that can be accessed to explore the data in question, each of which is accompanied by an equivalent JSON API. | [] | [] |
json_api:id2 | json_api | id2 | Table arguments | The Datasette table view takes a number of special query string arguments. | ["JSON API"] | [] |
spatialite:querying-polygons-using-within | spatialite | querying-polygons-using-within | Querying polygons using within() | The within() SQL function can be used to check if a point is within a geometry: select name from places where within(GeomFromText('POINT(-3.1724366 51.4704448)'), places.geom); The GeomFromText() function takes a string of well-known text. Note that the order used here is longitude then latitude . To run that same within() query in a way that benefits from the spatial index, use the following: select name from places where within(GeomFromText('POINT(-3.1724366 51.4704448)'), places.geom) and rowid in ( SELECT pkid FROM idx_places_geom where xmin < -3.1724366 and xmax > -3.1724366 and ymin < 51.4704448 and ymax > 51.4704448 ); | ["SpatiaLite"] | [] |
csv_export:streaming-all-records | csv_export | streaming-all-records | Streaming all records | The stream all rows option is designed to be as efficient as possible - under the hood it takes advantage of Python 3 asyncio capabilities and Datasette's efficient pagination to stream back the full CSV file. Since databases can get pretty large, by default this option is capped at 100MB - if a table returns more than 100MB of data the last line of the CSV will be a truncation error message. You can increase or remove this limit using the max_csv_mb config setting. You can also disable the CSV export feature entirely using allow_csv_stream . | ["CSV export"] | [] |
spatialite:importing-shapefiles-into-spatialite | spatialite | importing-shapefiles-into-spatialite | Importing shapefiles into SpatiaLite | The shapefile format is a common format for distributing geospatial data. You can use the spatialite command-line tool to create a new database table from a shapefile. Try it now with the North America shapefile available from the University of North Carolina Global River Database project. Download the file and unzip it (this will create files called narivs.dbf , narivs.prj , narivs.shp and narivs.shx in the current directory), then run the following: $ spatialite rivers-database.db SpatiaLite version ..: 4.3.0a Supported Extensions: ... spatialite> .loadshp narivs rivers CP1252 23032 ======== Loading shapefile at 'narivs' into SQLite table 'rivers' ... Inserted 467973 rows into 'rivers' from SHAPEFILE This will load the data from the narivs shapefile into a new database table called rivers . Exit out of spatialite (using Ctrl+D ) and run Datasette against your new database like this: datasette rivers-database.db \ --load-extension=/usr/local/lib/mod_spatialite.dylib If you browse to http://localhost:8001/rivers-database/rivers you will see the new table... but the Geometry column will contain unreadable binary data (SpatiaLite uses a custom format based on WKB ). The easiest way to turn this into semi-readable data is to use the SpatiaLite AsGeoJSON function. Try the following using the SQL query interface at http://localhost:8001/rivers-database : select *, AsGeoJSON(Geometry) from rivers limit 10; This will give you back an additional column of GeoJSON. You can copy and paste GeoJSON from this column into the debugging tool at geojson.io to visualize it on a map. To see a more interesting example, try ordering the records with the longest geometry first. Since there are 467,000 rows in the table you will first need to increase the SQL time limit imposed by Datasette: datasette rivers-database.db \ --load-extension=/us… | ["SpatiaLite"] | [{"href": "https://en.wikipedia.org/wiki/Shapefile", "label": "shapefile format"}, {"href": "http://gaia.geosci.unc.edu/rivers/", "label": "Global River Database"}, {"href": "https://www.gaia-gis.it/gaia-sins/BLOB-Geometry.html", "label": "a custom format based on WKB"}, {"href": "https://geojson.io/", "label": "geojson.io"}] |
changelog:plugin-hooks-and-internals | changelog | plugin-hooks-and-internals | Plugin hooks and internals | The prepare_jinja2_environment(env, datasette) plugin hook now accepts an optional datasette argument. Hook implementations can also now return an async function which will be awaited automatically. ( #1809 ) Database(is_mutable=) now defaults to True . ( #1808 ) The datasette.check_visibility() method now accepts an optional permissions= list, allowing it to take multiple permissions into account at once when deciding if something should be shown as public or private. This has been used to correctly display padlock icons in more places in the Datasette interface. ( #1829 ) Datasette no longer enforces upper bounds on its dependencies. ( #1800 ) | ["Changelog", "0.63 (2022-10-27)"] | [{"href": "https://github.com/simonw/datasette/issues/1809", "label": "#1809"}, {"href": "https://github.com/simonw/datasette/issues/1808", "label": "#1808"}, {"href": "https://github.com/simonw/datasette/issues/1829", "label": "#1829"}, {"href": "https://github.com/simonw/datasette/issues/1800", "label": "#1800"}] |
changelog:new-plugin-hook-extra-template-vars | changelog | new-plugin-hook-extra-template-vars | New plugin hook: extra_template_vars | The extra_template_vars(template, database, table, columns, view_name, request, datasette) plugin hook allows plugins to inject their own additional variables into the Datasette template context. This can be used in conjunction with custom templates to customize the Datasette interface. datasette-auth-github uses this hook to add custom HTML to the new top navigation bar (which is designed to be modified by plugins, see #540 ). | ["Changelog", "0.29 (2019-07-07)"] | [{"href": "https://github.com/simonw/datasette-auth-github", "label": "datasette-auth-github"}, {"href": "https://github.com/simonw/datasette/issues/540", "label": "#540"}] |
changelog:log-out | changelog | log-out | Log out | The ds_actor cookie can be used by plugins (or by Datasette's --root mechanism ) to authenticate users. The new /-/logout page provides a way to clear that cookie. A "Log out" button now shows in the global navigation provided the user is authenticated using the ds_actor cookie. ( #840 ) | ["Changelog", "0.45 (2020-07-01)"] | [{"href": "https://github.com/simonw/datasette/issues/840", "label": "#840"}] |
internals:database-results | internals | database-results | Results | The db.execute() method returns a single Results object. This can be used to access the rows returned by the query. Iterating over a Results object will yield SQLite Row objects . Each of these can be treated as a tuple or can be accessed using row["column"] syntax: info = [] results = await db.execute("select name from sqlite_master") for row in results: info.append(row["name"]) The Results object also has the following properties and methods: .truncated - boolean Indicates if this query was truncated - if it returned more results than the specified page_size . If this is true then the results object will only provide access to the first page_size rows in the query result. You can disable truncation by passing truncate=False to the db.query() method. .columns - list of strings A list of column names returned by the query. .rows - list of sqlite3.Row This property provides direct access to the list of rows returned by the database. You can access specific rows by index using results.rows[0] . .first() - row or None Returns the first row in the results, or None if no rows were returned. .single_value() Returns the value of the first column of the first row of results - but only if the query returned a single row with… | ["Internals for plugins", "Database class"] | [{"href": "https://docs.python.org/3/library/sqlite3.html#row-objects", "label": "Row objects"}] |
installation:loading-spatialite | installation | loading-spatialite | Loading SpatiaLite | The datasetteproject/datasette image includes a recent version of the SpatiaLite extension for SQLite. To load and enable that module, use the following command: docker run -p 8001:8001 -v `pwd`:/mnt \ datasetteproject/datasette \ datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db \ --load-extension=spatialite You can confirm that SpatiaLite is successfully loaded by visiting http://127.0.0.1:8001/-/versions | ["Installation", "Advanced installation options", "Using Docker"] | [{"href": "http://127.0.0.1:8001/-/versions", "label": "http://127.0.0.1:8001/-/versions"}] |
internals:internals-utils | internals | internals-utils | The datasette.utils module | The datasette.utils module contains various utility functions used by Datasette. As a general rule you should consider anything in this module to be unstable - functions and classes here could change without warning or be removed entirely between Datasette releases, without being mentioned in the release notes. The exception to this rule is anythang that is documented here. If you find a need for an undocumented utility function in your own work, consider opening an issue requesting that the function you are using be upgraded to documented and supported status. | ["Internals for plugins"] | [{"href": "https://github.com/simonw/datasette/issues/new", "label": "opening an issue"}] |
internals:internals-datasette-urls | internals | internals-datasette-urls | datasette.urls | The datasette.urls object contains methods for building URLs to pages within Datasette. Plugins should use this to link to pages, since these methods take into account any base_url configuration setting that might be in effect. datasette.urls.instance(format=None) Returns the URL to the Datasette instance root page. This is usually "/" . datasette.urls.path(path, format=None) Takes a path and returns the full path, taking base_url into account. For example, datasette.urls.path("-/logout") will return the path to the logout page, which will be "/-/logout" by default or /prefix-path/-/logout if base_url is set to /prefix-path/ datasette.urls.logout() Returns the URL to the logout page, usually "/-/logout" datasette.urls.static(path) Returns the URL of one of Datasette's default static assets, for example "/-/static/app.css" datasette.urls.static_plugins(plugin_name, path) Returns the URL of one of the static assets belonging to a plugin. datasette.urls.static_plugins("datasette_cluster_map", "datasette-cluster-map.js") would return "/-/static-plugins/datasette_cluster_map/datasette-cluster-map.js" datasette.urls.static(path) … | ["Internals for plugins", "Datasette class"] | [] |
custom_templates:publishing-static-assets | custom_templates | publishing-static-assets | Publishing static assets | The datasette publish command can be used to publish your static assets, using the same syntax as above: $ datasette publish cloudrun mydb.db --static assets:static-files/ This will upload the contents of the static-files/ directory as part of the deployment, and configure Datasette to correctly serve the assets from /assets/ . | ["Custom pages and templates", "Custom CSS and JavaScript"] | [] |
plugins:deploying-plugins-using-datasette-publish | plugins | deploying-plugins-using-datasette-publish | Deploying plugins using datasette publish | The datasette publish and datasette package commands both take an optional --install argument. You can use this one or more times to tell Datasette to pip install specific plugins as part of the process: datasette publish cloudrun mydb.db --install=datasette-vega You can use the name of a package on PyPI or any of the other valid arguments to pip install such as a URL to a .zip file: datasette publish cloudrun mydb.db \ --install=https://url-to-my-package.zip | ["Plugins", "Installing plugins"] | [] |
settings:setting-publish-secrets | settings | setting-publish-secrets | Using secrets with datasette publish | The datasette publish and datasette package commands both generate a secret for you automatically when Datasette is deployed. This means that every time you deploy a new version of a Datasette project, a new secret will be generated. This will cause signed cookies to become invalid on every fresh deploy. You can fix this by creating a secret that will be used for multiple deploys and passing it using the --secret option: datasette publish cloudrun mydb.db --service=my-service --secret=cdb19e94283a20f9d42cca5 | ["Settings"] | [] |
changelog:id50 | changelog | id50 | 0.47.3 (2020-08-15) | The datasette --get command-line mechanism now ensures any plugins using the startup() hook are correctly executed. ( #934 ) | ["Changelog"] | [{"href": "https://github.com/simonw/datasette/issues/934", "label": "#934"}] |
changelog:id67 | changelog | id67 | 0.36 (2020-02-21) | The datasette object passed to plugins now has API documentation: Datasette class . ( #576 ) New methods on datasette : .add_database() and .remove_database() - documentation . ( #671 ) prepare_connection() plugin hook now takes optional datasette and database arguments - prepare_connection(conn, database, datasette) . ( #678 ) Added three new plugins and one new conversion tool to the The Datasette Ecosystem . | ["Changelog"] | [{"href": "https://github.com/simonw/datasette/issues/576", "label": "#576"}, {"href": "https://github.com/simonw/datasette/issues/671", "label": "#671"}, {"href": "https://github.com/simonw/datasette/issues/678", "label": "#678"}] |
cli-reference:id1 | cli-reference | id1 | CLI reference | The datasette CLI tool provides a number of commands. Running datasette without specifying a command runs the default command, datasette serve . See datasette serve for the full list of options for that command. [[[cog from datasette import cli from click.testing import CliRunner import textwrap def help(args): title = "datasette " + " ".join(args) cog.out("\n::\n\n") result = CliRunner().invoke(cli.cli, args) output = result.output.replace("Usage: cli ", "Usage: datasette ") cog.out(textwrap.indent(output, ' ')) cog.out("\n\n") ]]] [[[end]]] | [] | [] |
contributing:contributing-formatting-blacken-docs | contributing | contributing-formatting-blacken-docs | blacken-docs | The blacken-docs command applies Black formatting rules to code examples in the documentation. Run it like this: blacken-docs -l 60 docs/*.rst | ["Contributing", "Code formatting"] | [{"href": "https://pypi.org/project/blacken-docs/", "label": "blacken-docs"}] |
changelog:running-datasette-behind-a-proxy | changelog | running-datasette-behind-a-proxy | Running Datasette behind a proxy | The base_url configuration option is designed to help run Datasette on a specific path behind a proxy - for example if you want to run an instance of Datasette at /my-datasette/ within your existing site's URL hierarchy, proxied behind nginx or Apache. Support for this configuration option has been greatly improved ( #1023 ), and guidelines for using it are now available in a new documentation section on Running Datasette behind a proxy . ( #1027 ) | ["Changelog", "0.51 (2020-10-31)"] | [{"href": "https://github.com/simonw/datasette/issues/1023", "label": "#1023"}, {"href": "https://github.com/simonw/datasette/issues/1027", "label": "#1027"}] |
changelog:new-plugin-hook-asgi-wrapper | changelog | new-plugin-hook-asgi-wrapper | New plugin hook: asgi_wrapper | The asgi_wrapper(datasette) plugin hook allows plugins to entirely wrap the Datasette ASGI application in their own ASGI middleware. ( #520 ) Two new plugins take advantage of this hook: datasette-auth-github adds a authentication layer: users will have to sign in using their GitHub account before they can view data or interact with Datasette. You can also use it to restrict access to specific GitHub users, or to members of specified GitHub organizations or teams . datasette-cors allows you to configure CORS headers for your Datasette instance. You can use this to enable JavaScript running on a whitelisted set of domains to make fetch() calls to the JSON API provided by your Datasette instance. | ["Changelog", "0.29 (2019-07-07)"] | [{"href": "https://github.com/simonw/datasette/issues/520", "label": "#520"}, {"href": "https://github.com/simonw/datasette-auth-github", "label": "datasette-auth-github"}, {"href": "https://help.github.com/en/articles/about-organizations", "label": "organizations"}, {"href": "https://help.github.com/en/articles/organizing-members-into-teams", "label": "teams"}, {"href": "https://github.com/simonw/datasette-cors", "label": "datasette-cors"}, {"href": "https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS", "label": "CORS headers"}] |
spatialite:id1 | spatialite | id1 | SpatiaLite | The SpatiaLite module for SQLite adds features for handling geographic and spatial data. For an example of what you can do with it, see the tutorial Building a location to time zone API with SpatiaLite . To use it with Datasette, you need to install the mod_spatialite dynamic library. This can then be loaded into Datasette using the --load-extension command-line option. Datasette can look for SpatiaLite in common installation locations if you run it like this: datasette --load-extension=spatialite --setting default_allow_sql off If SpatiaLite is in another location, use the full path to the extension instead: datasette --setting default_allow_sql off \ --load-extension=/usr/local/lib/mod_spatialite.dylib | [] | [{"href": "https://www.gaia-gis.it/fossil/libspatialite/index", "label": "SpatiaLite module"}, {"href": "https://datasette.io/tutorials/spatialite", "label": "Building a location to time zone API with SpatiaLite"}] |
changelog:improved-support-for-spatialite | changelog | improved-support-for-spatialite | Improved support for SpatiaLite | The SpatiaLite module for SQLite adds robust geospatial features to the database. Getting SpatiaLite working can be tricky, especially if you want to use the most recent alpha version (with support for K-nearest neighbor). Datasette now includes extensive documentation on SpatiaLite , and thanks to Ravi Kotecha our GitHub repo includes a Dockerfile that can build the latest SpatiaLite and configure it for use with Datasette. The datasette publish and datasette package commands now accept a new --spatialite argument which causes them to install and configure SpatiaLite as part of the container they deploy. | ["Changelog", "0.23 (2018-06-18)"] | [{"href": "https://www.gaia-gis.it/fossil/libspatialite/index", "label": "SpatiaLite module"}, {"href": "https://github.com/r4vi", "label": "Ravi Kotecha"}, {"href": "https://github.com/simonw/datasette/blob/master/Dockerfile", "label": "Dockerfile"}] |
internals:internals-response | internals | internals-response | Response class | The Response class can be returned from view functions that have been registered using the register_routes(datasette) hook. The Response() constructor takes the following arguments: body - string The body of the response. status - integer (optional) The HTTP status - defaults to 200. headers - dictionary (optional) A dictionary of extra HTTP headers, e.g. {"x-hello": "world"} . content_type - string (optional) The content-type for the response. Defaults to text/plain . For example: from datasette.utils.asgi import Response response = Response( "<xml>This is XML</xml>", content_type="application/xml; charset=utf-8", ) The quickest way to create responses is using the Response.text(...) , Response.html(...) , Response.json(...) or Response.redirect(...) helper methods: from datasette.utils.asgi import Response html_response = Response.html("This is HTML") json_response = Response.json({"this_is": "json"}) text_response = Response.text( "This will become utf-8 encoded text" ) # Redirects are served as 302, unless you pass status=301: redirect_response = Response.redirect( "https://latest.datasette.io/" ) Each of these responses will use the correct corresponding content-type - text/html; charset=utf-8 , application/json; charset=utf-8 or text/plain; charset=utf-8 respectively. Each of the helper methods take optional status= and headers= argument… | ["Internals for plugins"] | [] |
changelog:id64 | changelog | id64 | 0.38 (2020-03-08) | The Docker build of Datasette now uses SQLite 3.31.1, upgraded from 3.26. ( #695 ) datasette publish cloudrun now accepts an optional --memory=2Gi flag for setting the Cloud Run allocated memory to a value other than the default (256Mi). ( #694 ) Fixed bug where templates that shipped with plugins were sometimes not being correctly loaded. ( #697 ) | ["Changelog"] | [{"href": "https://hub.docker.com/r/datasetteproject/datasette", "label": "Docker build"}, {"href": "https://github.com/simonw/datasette/issues/695", "label": "#695"}, {"href": "https://github.com/simonw/datasette/issues/694", "label": "#694"}, {"href": "https://github.com/simonw/datasette/issues/697", "label": "#697"}] |
internals:database-constructor | internals | database-constructor | Database(ds, path=None, is_mutable=True, is_memory=False, memory_name=None) | The Database() constructor can be used by plugins, in conjunction with .add_database(db, name=None, route=None) , to create and register new databases. The arguments are as follows: ds - Datasette class (required) The Datasette instance you are attaching this database to. path - string Path to a SQLite database file on disk. is_mutable - boolean Set this to False to cause Datasette to open the file in immutable mode. is_memory - boolean Use this to create non-shared memory connections. memory_name - string or None Use this to create a named in-memory database. Unlike regular memory databases these can be accessed by multiple threads and will persist an changes made to them for the lifetime of the Datasette server process. The first argument is the datasette instance you are attaching to, the second is a path= , then is_mutable and is_memory are both optional arguments. | ["Internals for plugins", "Database class"] | [] |
internals:internals-database-introspection | internals | internals-database-introspection | Database introspection | The Database class also provides properties and methods for introspecting the database. db.name - string The name of the database - usually the filename without the .db prefix. db.size - integer The size of the database file in bytes. 0 for :memory: databases. db.mtime_ns - integer or None The last modification time of the database file in nanoseconds since the epoch. None for :memory: databases. db.is_mutable - boolean Is this database mutable, and allowed to accept writes? db.is_memory - boolean Is this database an in-memory database? await db.attached_databases() - list of named tuples Returns a list of additional databases that have been connected to this database using the SQLite ATTACH command. Each named tuple has fields seq , name and file . await db.table_exists(table) - boolean Check if a table called table exists. await db.table_names() - list of strings … | ["Internals for plugins", "Database class"] | [] |
authentication:allowdebugview | authentication | allowdebugview | The /-/allow-debug tool | The /-/allow-debug tool lets you try out different "action" blocks against different "actor" JSON objects. You can try that out here: https://latest.datasette.io/-/allow-debug | ["Authentication and permissions", "Permissions"] | [{"href": "https://latest.datasette.io/-/allow-debug", "label": "https://latest.datasette.io/-/allow-debug"}] |
binary_data:binary-linking | binary_data | binary-linking | Linking to binary downloads | The .blob output format is used to return binary data. It requires a _blob_column= query string argument specifying which BLOB column should be downloaded, for example: https://latest.datasette.io/fixtures/binary_data/1.blob?_blob_column=data This output format can also be used to return binary data from an arbitrary SQL query. Since such queries do not specify an exact row, an additional ?_blob_hash= parameter can be used to specify the SHA-256 hash of the value that is being linked to. Consider the query select data from binary_data - demonstrated here . That page links to the binary value downloads. Those links look like this: https://latest.datasette.io/fixtures.blob?sql=select+data+from+binary_data&_blob_column=data&_blob_hash=f3088978da8f9aea479ffc7f631370b968d2e855eeb172bea7f6c7a04262bb6d These .blob links are also returned in the .csv exports Datasette provides for binary tables and queries, since the CSV format does not have a mechanism for representing binary data. | ["Binary data"] | [{"href": "https://latest.datasette.io/fixtures/binary_data/1.blob?_blob_column=data", "label": "https://latest.datasette.io/fixtures/binary_data/1.blob?_blob_column=data"}, {"href": "https://latest.datasette.io/fixtures?sql=select+data+from+binary_data", "label": "demonstrated here"}, {"href": "https://latest.datasette.io/fixtures.blob?sql=select+data+from+binary_data&_blob_column=data&_blob_hash=f3088978da8f9aea479ffc7f631370b968d2e855eeb172bea7f6c7a04262bb6d", "label": "https://latest.datasette.io/fixtures.blob?sql=select+data+from+binary_data&_blob_column=data&_blob_hash=f3088978da8f9aea479ffc7f631370b968d2e855eeb172bea7f6c7a04262bb6d"}] |
cli-reference:cli-datasette-get | cli-reference | cli-datasette-get | datasette --get | The --get option to datasette serve (or just datasette ) specifies the path to a page within Datasette and causes Datasette to output the content from that path without starting the web server. This means that all of Datasette's functionality can be accessed directly from the command-line. For example: $ datasette --get '/-/versions.json' | jq . { "python": { "version": "3.8.5", "full": "3.8.5 (default, Jul 21 2020, 10:48:26) \n[Clang 11.0.3 (clang-1103.0.32.62)]" }, "datasette": { "version": "0.46+15.g222a84a.dirty" }, "asgi": "3.0", "uvicorn": "0.11.8", "sqlite": { "version": "3.32.3", "fts_versions": [ "FTS5", "FTS4", "FTS3" ], "extensions": { "json1": null }, "compile_options": [ "COMPILER=clang-11.0.3", "ENABLE_COLUMN_METADATA", "ENABLE_FTS3", "ENABLE_FTS3_PARENTHESIS", "ENABLE_FTS4", "ENABLE_FTS5", "ENABLE_GEOPOLY", "ENABLE_JSON1", "ENABLE_PREUPDATE_HOOK", "ENABLE_RTREE", "ENABLE_SESSION", "MAX_VARIABLE_NUMBER=250000", "THREADSAFE=1" ] } } The exit code will be 0 if the request succeeds and 1 if the request produced an HTTP status code other than 200 - e.g. a 404 or 500 error. This lets you use datasette --get / to run tests against a Datasette application in a continuous integration environment such as GitHub Actions. | ["CLI reference", "datasette serve"] | [] |
full_text_search:full-text-search-table-view-api | full_text_search | full-text-search-table-view-api | The table page and table view API | Table views that support full-text search can be queried using the ?_search=TERMS query string parameter. This will run the search against content from all of the columns that have been included in the index. Try this example: fara.datasettes.com/fara/FARA_All_ShortForms?_search=manafort SQLite full-text search supports wildcards. This means you can easily implement prefix auto-complete by including an asterisk at the end of the search term - for example: /dbname/tablename/?_search=rob* This will return all records containing at least one word that starts with the letters rob . You can also run searches against just the content of a specific named column by using _search_COLNAME=TERMS - for example, this would search for just rows where the name column in the FTS index mentions Sarah : /dbname/tablename/?_search_name=Sarah | ["Full-text search"] | [{"href": "https://fara.datasettes.com/fara/FARA_All_ShortForms?_search=manafort", "label": "fara.datasettes.com/fara/FARA_All_ShortForms?_search=manafort"}] |
changelog:id32 | changelog | id32 | 0.55 (2021-02-18) | Support for cross-database SQL queries and built-in support for serving via HTTPS. The new --crossdb command-line option causes Datasette to attach up to ten database files to the same /_memory database connection. This enables cross-database SQL queries, including the ability to use joins and unions to combine data from tables that exist in different database files. See Cross-database queries for details. ( #283 ) --ssl-keyfile and --ssl-certfile options can be used to specify a TLS certificate, allowing Datasette to serve traffic over https:// without needing to run it behind a separate proxy. ( #1221 ) The /:memory: page has been renamed (and redirected) to /_memory for consistency with the new /_internal database introduced in Datasette 0.54. ( #1205 ) Added plugin testing documentation on Using pdb for errors thrown inside Datasette . ( #1207 ) The official Datasette Docker image now uses Python 3.7.10, applying the latest security fix for that Python version. ( #1235 ) | ["Changelog"] | [{"href": "https://github.com/simonw/datasette/issues/283", "label": "#283"}, {"href": "https://github.com/simonw/datasette/issues/1221", "label": "#1221"}, {"href": "https://github.com/simonw/datasette/issues/1205", "label": "#1205"}, {"href": "https://github.com/simonw/datasette/issues/1207", "label": "#1207"}, {"href": "https://hub.docker.com/r/datasetteproject/datasette", "label": "official Datasette Docker image"}, {"href": "https://www.python.org/downloads/release/python-3710/", "label": "the latest security fix"}, {"href": "https://github.com/simonw/datasette/issues/1235", "label": "#1235"}] |
spatialite:making-use-of-a-spatial-index | spatialite | making-use-of-a-spatial-index | Making use of a spatial index | SpatiaLite spatial indexes are R*Trees. They allow you to run efficient bounding box queries using a sub-select, with a similar pattern to that used for Searches using custom SQL . In the above example, the resulting index will be called idx_museums_point_geom . This takes the form of a SQLite virtual table. You can inspect its contents using the following query: select * from idx_museums_point_geom limit 10; Here's a live example: timezones-api.datasette.io/timezones/idx_timezones_Geometry pkid xmin xmax ymin ymax 1 -8.601725578308105 -2.4930307865142822 4.162120819091797 10.74019718170166 2 … | ["SpatiaLite"] | [{"href": "https://timezones-api.datasette.io/timezones/idx_timezones_Geometry", "label": "timezones-api.datasette.io/timezones/idx_timezones_Geometry"}] |
spatialite:installing-spatialite-on-linux | spatialite | installing-spatialite-on-linux | Installing SpatiaLite on Linux | SpatiaLite is packaged for most Linux distributions. apt install spatialite-bin libsqlite3-mod-spatialite Depending on your distribution, you should be able to run Datasette something like this: datasette --load-extension=/usr/lib/x86_64-linux-gnu/mod_spatialite.so If you are unsure of the location of the module, try running locate mod_spatialite and see what comes back. | ["SpatiaLite", "Installation"] | [] |
sql_queries:fragment | sql_queries | fragment | fragment | Some plugins, such as datasette-vega , can be configured by including additional data in the fragment hash of the URL - the bit that comes after a # symbol. You can set a default fragment hash that will be included in the link to the canned query from the database index page using the "fragment" key. This example demonstrates both fragment and hide_sql : { "databases": { "fixtures": { "queries": { "neighborhood_search": { "sql": "select neighborhood, facet_cities.name, state\nfrom facetable join facet_cities on facetable.city_id = facet_cities.id\nwhere neighborhood like '%' || :text || '%' order by neighborhood;", "fragment": "fragment-goes-here", "hide_sql": true } } } } } See here for a demo of this in action. | ["Running SQL queries", "Canned queries", "Additional canned query options"] | [{"href": "https://github.com/simonw/datasette-vega", "label": "datasette-vega"}, {"href": "https://latest.datasette.io/fixtures#queries", "label": "See here"}] |
contributing:contributing-documentation-cog | contributing | contributing-documentation-cog | Running Cog | Some pages of documentation (in particular the CLI reference ) are automatically updated using Cog . To update these pages, run the following command: cog -r docs/*.rst | ["Contributing", "Editing and building the documentation"] | [{"href": "https://github.com/nedbat/cog", "label": "Cog"}] |
deploying:deploying-buildpacks | deploying | deploying-buildpacks | Deploying using buildpacks | Some hosting providers such as Heroku , DigitalOcean App Platform and Scalingo support the Buildpacks standard for deploying Python web applications. Deploying Datasette on these platforms requires two files: requirements.txt and Procfile . The requirements.txt file lets the platform know which Python packages should be installed. It should contain datasette at a minimum, but can also list any Datasette plugins you wish to install - for example: datasette datasette-vega The Procfile lets the hosting platform know how to run the command that serves web traffic. It should look like this: web: datasette . -h 0.0.0.0 -p $PORT --cors The $PORT environment variable is provided by the hosting platform. --cors enables CORS requests from JavaScript running on other websites to your domain - omit this if you don't want to allow CORS. You can add additional Datasette Settings options here too. These two files should be enough to deploy Datasette on any host that supports buildpacks. Datasette will serve any SQLite files that are included in the root directory of the application. If you want to build SQLite files or download them as part of the deployment process you can do so using a bin/post_compile file. For example, the following bin/post_compile will download an example database that will then be served by Datasette: wget https://fivethirtyeight.datasettes.com/fivethirtyeight.db simonw/buildpack-datasette-demo is an example GitHub repository showing a Datasette configuration that can be deployed to a buildpack-supporting host. | ["Deploying Datasette"] | [{"href": "https://www.heroku.com/", "label": "Heroku"}, {"href": "https://www.digitalocean.com/docs/app-platform/", "label": "DigitalOcean App Platform"}, {"href": "https://scalingo.com/", "label": "Scalingo"}, {"href": "https://buildpacks.io/", "label": "Buildpacks standard"}, {"href": "https://github.com/simonw/buildpack-datasette-demo", "label": "simonw/buildpack-datasette-demo"}] |
changelog:csrf-protection | changelog | csrf-protection | CSRF protection | Since writable canned queries are built using POST forms, Datasette now ships with CSRF protection ( #798 ). This applies automatically to any POST request, which means plugins need to include a csrftoken in any POST forms that they render. They can do that like so: <input type="hidden" name="csrftoken" value="{{ csrftoken() }}"> | ["Changelog", "0.44 (2020-06-11)"] | [{"href": "https://github.com/simonw/datasette/issues/798", "label": "#798"}] |
introspection:jsondataview-versions | introspection | jsondataview-versions | /-/versions | Shows the version of Datasette, Python and SQLite. Versions example : { "datasette": { "version": "0.60" }, "python": { "full": "3.8.12 (default, Dec 21 2021, 10:45:09) \n[GCC 10.2.1 20210110]", "version": "3.8.12" }, "sqlite": { "extensions": { "json1": null }, "fts_versions": [ "FTS5", "FTS4", "FTS3" ], "compile_options": [ "COMPILER=gcc-6.3.0 20170516", "ENABLE_FTS3", "ENABLE_FTS4", "ENABLE_FTS5", "ENABLE_JSON1", "ENABLE_RTREE", "THREADSAFE=1" ], "version": "3.37.0" } } | ["Introspection"] | [{"href": "https://latest.datasette.io/-/versions", "label": "Versions example"}] |
introspection:jsondataview-actor | introspection | jsondataview-actor | /-/actor | Shows the currently authenticated actor. Useful for debugging Datasette authentication plugins. { "actor": { "id": 1, "username": "some-user" } } | ["Introspection"] | [] |
introspection:jsondataview-metadata | introspection | jsondataview-metadata | /-/metadata | Shows the contents of the metadata.json file that was passed to datasette serve , if any. Metadata example : { "license": "CC Attribution 4.0 License", "license_url": "http://creativecommons.org/licenses/by/4.0/", "source": "fivethirtyeight/data on GitHub", "source_url": "https://github.com/fivethirtyeight/data", "title": "Five Thirty Eight", "databases": { } } | ["Introspection"] | [{"href": "https://fivethirtyeight.datasettes.com/-/metadata", "label": "Metadata example"}] |
introspection:jsondataview-config | introspection | jsondataview-config | /-/settings | Shows the Settings for this instance of Datasette. Settings example : { "default_facet_size": 30, "default_page_size": 100, "facet_suggest_time_limit_ms": 50, "facet_time_limit_ms": 1000, "max_returned_rows": 1000, "sql_time_limit_ms": 1000 } | ["Introspection"] | [{"href": "https://fivethirtyeight.datasettes.com/-/settings", "label": "Settings example"}] |
introspection:jsondataview-threads | introspection | jsondataview-threads | /-/threads | Shows details of threads and asyncio tasks. Threads example : { "num_threads": 2, "threads": [ { "daemon": false, "ident": 4759197120, "name": "MainThread" }, { "daemon": true, "ident": 123145319682048, "name": "Thread-1" }, ], "num_tasks": 3, "tasks": [ "<Task pending coro=<RequestResponseCycle.run_asgi() running at uvicorn/protocols/http/httptools_impl.py:385> cb=[set.discard()]>", "<Task pending coro=<Server.serve() running at uvicorn/main.py:361> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x10365c3d0>()]> cb=[run_until_complete.<locals>.<lambda>()]>", "<Task pending coro=<LifespanOn.main() running at uvicorn/lifespan/on.py:48> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x10364f050>()]>>" ] } | ["Introspection"] | [{"href": "https://latest.datasette.io/-/threads", "label": "Threads example"}] |
introspection:jsondataview-databases | introspection | jsondataview-databases | /-/databases | Shows currently attached databases. Databases example : [ { "hash": null, "is_memory": false, "is_mutable": true, "name": "fixtures", "path": "fixtures.db", "size": 225280 } ] | ["Introspection"] | [{"href": "https://latest.datasette.io/-/databases", "label": "Databases example"}] |
introspection:jsondataview-plugins | introspection | jsondataview-plugins | /-/plugins | Shows a list of currently installed plugins and their versions. Plugins example : [ { "name": "datasette_cluster_map", "static": true, "templates": false, "version": "0.10", "hooks": ["extra_css_urls", "extra_js_urls", "extra_body_script"] } ] Add ?all=1 to include details of the default plugins baked into Datasette. | ["Introspection"] | [{"href": "https://san-francisco.datasettes.com/-/plugins", "label": "Plugins example"}] |
cli-reference:cli-help-publish-help | cli-reference | cli-help-publish-help | datasette publish | Shows a list of available deployment targets for publishing data with Datasette. Additional deployment targets can be added by plugins that use the publish_subcommand(publish) hook. [[[cog help(["publish", "--help"]) ]]] Usage: datasette publish [OPTIONS] COMMAND [ARGS]... Publish specified SQLite database files to the internet along with a Datasette-powered interface and API Options: --help Show this message and exit. Commands: cloudrun Publish databases to Datasette running on Cloud Run heroku Publish databases to Datasette running on Heroku [[[end]]] | ["CLI reference"] | [] |
changelog:id37 | changelog | id37 | 0.52.4 (2020-12-05) | Show pysqlite3 version on /-/versions , if installed. ( #1125 ) Errors output by Datasette (e.g. for invalid SQL queries) now go to stderr , not stdout . ( #1131 ) Fix for a startup error on windows caused by unnecessary from os import EX_CANTCREAT - thanks, Abdussamet Koçak. ( #1094 ) | ["Changelog"] | [{"href": "https://github.com/coleifer/pysqlite3", "label": "pysqlite3"}, {"href": "https://github.com/simonw/datasette/issues/1125", "label": "#1125"}, {"href": "https://github.com/simonw/datasette/issues/1131", "label": "#1131"}, {"href": "https://github.com/simonw/datasette/issues/1094", "label": "#1094"}] |
settings:setting-default-allow-sql | settings | setting-default-allow-sql | default_allow_sql | Should users be able to execute arbitrary SQL queries by default? Setting this to off causes permission checks for execute-sql to fail by default. datasette mydatabase.db --setting default_allow_sql off There are two ways to achieve this: the other is to add "allow_sql": false to your metadata.json file, as described in Controlling the ability to execute arbitrary SQL . This setting offers a more convenient way to do this. | ["Settings", "Settings"] | [] |
settings:setting-allow-download | settings | setting-allow-download | allow_download | Should users be able to download the original SQLite database using a link on the database index page? This is turned on by default. However, databases can only be downloaded if they are served in immutable mode and not in-memory. If downloading is unavailable for either of these reasons, the download link is hidden even if allow_download is on. To disable database downloads, use the following: datasette mydatabase.db --setting allow_download off | ["Settings", "Settings"] | [] |
settings:setting-suggest-facets | settings | setting-suggest-facets | suggest_facets | Should Datasette calculate suggested facets? On by default, turn this off like so: datasette mydatabase.db --setting suggest_facets off | ["Settings", "Settings"] | [] |
binary_data:binary-plugins | binary_data | binary-plugins | Binary plugins | Several Datasette plugins are available that change the way Datasette treats binary data. datasette-render-binary modifies Datasette's default interface to show an automatic guess at what type of binary data is being stored, along with a visual representation of the binary value that displays ASCII strings directly in the interface. datasette-render-images detects common image formats and renders them as images directly in the Datasette interface. datasette-media allows Datasette interfaces to be configured to serve binary files from configured SQL queries, and includes the ability to resize images directly before serving them. | ["Binary data"] | [{"href": "https://github.com/simonw/datasette-render-binary", "label": "datasette-render-binary"}, {"href": "https://github.com/simonw/datasette-render-images", "label": "datasette-render-images"}, {"href": "https://github.com/simonw/datasette-media", "label": "datasette-media"}] |
settings:setting-cache-size-kb | settings | setting-cache-size-kb | cache_size_kb | Sets the amount of memory SQLite uses for its per-connection cache , in KB. datasette mydatabase.db --setting cache_size_kb 5000 | ["Settings", "Settings"] | [{"href": "https://www.sqlite.org/pragma.html#pragma_cache_size", "label": "per-connection cache"}] |
changelog:id48 | changelog | id48 | 0.49 (2020-09-14) | See also Datasette 0.49: The annotated release notes . Writable canned queries now expose a JSON API, see JSON API for writable canned queries . ( #880 ) New mechanism for defining page templates with custom path parameters - a template file called pages/about/{slug}.html will be used to render any requests to /about/something . See Path parameters for pages . ( #944 ) register_output_renderer() render functions can now return a Response . ( #953 ) New --upgrade option for datasette install . ( #945 ) New datasette --pdb option. ( #962 ) datasette --get exit code now reflects the internal HTTP status code. ( #947 ) New raise_404() template function for returning 404 errors. ( #964 ) datasette publish heroku now deploys using Python 3.8.5 Upgraded CodeMirror to 5.57.0. ( #948 ) Upgraded code style to Black 20.8b1. ( #958 ) Fixed bug where selected facets were not correctly persisted in hidden form fields on the table page. ( #963 ) Renamed the default error template from 500.html to error.html . Custom error pages are now documented, see Custom error pages . ( #965 ) | ["Changelog"] | [{"href": "https://simonwillison.net/2020/Sep/15/datasette-0-49/", "label": "Datasette 0.49: The annotated release notes"}, {"href": "https://github.com/simonw/datasette/issues/880", "label": "#880"}, {"href": "https://github.com/simonw/datasette/issues/944", "label": "#944"}, {"href": "https://github.com/simonw/datasette/issues/953", "label": "#953"}, {"href": "https://github.com/simonw/datasette/issues/945", "label": "#945"}, {"href": "https://github.com/simonw/datasette/issues/962", "label": "#962"}, {"href": "https://github.com/simonw/datasette/issues/947", "label": "#947"}, {"href": "https://github.com/simonw/datasette/issues/964", "label": "#964"}, {"href": "https://codemirror.net/", "label": "CodeMirror"}, {"href": "https://github.com/simonw/datasette/issues/948", "label": "#948"}, {"href": "https://github.com/simonw/datasette/issues/958", "label": "#958"}, {"href": "https://github.com/simonw/datasette/issues/963", "label": "#963"}, {"href": "https://github.com/simonw/datasette/issues/965", "label": "#965"}] |
changelog:id55 | changelog | id55 | 0.45 (2020-07-01) | See also Datasette 0.45: The annotated release notes . Magic parameters for canned queries, a log out feature, improved plugin documentation and four new plugin hooks. | ["Changelog"] | [{"href": "https://simonwillison.net/2020/Jul/1/datasette-045/", "label": "Datasette 0.45: The annotated release notes"}] |
changelog:id57 | changelog | id57 | 0.44 (2020-06-11) | See also Datasette 0.44: The annotated release notes . Authentication and permissions, writable canned queries, flash messages, new plugin hooks and more. | ["Changelog"] | [{"href": "https://simonwillison.net/2020/Jun/12/annotated-release-notes/", "label": "Datasette 0.44: The annotated release notes"}] |
cli-reference:cli-help-publish-heroku-help | cli-reference | cli-help-publish-heroku-help | datasette publish heroku | See Publishing to Heroku . [[[cog help(["publish", "heroku", "--help"]) ]]] Usage: datasette publish heroku [OPTIONS] [FILES]... Publish databases to Datasette running on Heroku Options: -m, --metadata FILENAME Path to JSON/YAML file containing metadata to publish --extra-options TEXT Extra options to pass to datasette serve --branch TEXT Install datasette from a GitHub branch e.g. main --template-dir DIRECTORY Path to directory containing custom templates --plugins-dir DIRECTORY Path to directory containing custom plugins --static MOUNT:DIRECTORY Serve static files from this directory at /MOUNT/... --install TEXT Additional packages (e.g. plugins) to install --plugin-secret <TEXT TEXT TEXT>... Secrets to pass to plugins, e.g. --plugin- secret datasette-auth-github client_id xxx --version-note TEXT Additional note to show on /-/versions --secret TEXT Secret used for signing secure values, such as signed cookies --title TEXT Title for metadata --license TEXT License label for metadata --license_url TEXT License URL for metadata --source TEXT Source label for metadata --source_url TEXT Source URL for metadata --about TEXT About label for metadata --about_url TEXT About URL for metadata -n, --name TEXT Application name to use when deploying --tar TEXT --tar option to pass to Heroku, e.g. --tar=/usr/local/bin/gtar --generate-dir DIRECTORY Output generated application files and stop without deploying --h… | ["CLI reference"] | [] |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [sections] ( [id] TEXT PRIMARY KEY, [page] TEXT, [ref] TEXT, [title] TEXT, [content] TEXT, [breadcrumbs] TEXT, [references] TEXT );