{"rowid": 1, "title": "Writing plugins", "content": "You can write one-off plugins that apply to just one Datasette instance, or you can write plugins which can be installed using pip and can be shipped to the Python Package Index ( PyPI ) for other people to install. \n Want to start by looking at an example? The Datasette plugins directory lists more than 90 open source plugins with code you can explore. The plugin hooks page includes links to example plugins for each of the documented hooks.", "sections_fts": 186, "rank": null} {"rowid": 2, "title": "Writing one-off plugins", "content": "The quickest way to start writing a plugin is to create a my_plugin.py file and drop it into your plugins/ directory. Here is an example plugin, which adds a new custom SQL function called hello_world() which takes no arguments and returns the string Hello world! . \n from datasette import hookimpl\n\n\n@hookimpl\ndef prepare_connection(conn):\n conn.create_function(\n \"hello_world\", 0, lambda: \"Hello world!\"\n ) \n If you save this in plugins/my_plugin.py you can then start Datasette like this: \n datasette serve mydb.db --plugins-dir=plugins/ \n Now you can navigate to http://localhost:8001/mydb and run this SQL: \n select hello_world(); \n To see the output of your plugin.", "sections_fts": 186, "rank": null} {"rowid": 3, "title": "Starting an installable plugin using cookiecutter", "content": "Plugins that can be installed should be written as Python packages using a setup.py file. \n The quickest way to start writing one an installable plugin is to use the datasette-plugin cookiecutter template. This creates a new plugin structure for you complete with an example test and GitHub Actions workflows for testing and publishing your plugin. \n Install cookiecutter and then run this command to start building a plugin using the template: \n cookiecutter gh:simonw/datasette-plugin \n Read a cookiecutter template for writing Datasette plugins for more information about this template.", "sections_fts": 186, "rank": null} {"rowid": 4, "title": "Packaging a plugin", "content": "Plugins can be packaged using Python setuptools. You can see an example of a packaged plugin at https://github.com/simonw/datasette-plugin-demos \n The example consists of two files: a setup.py file that defines the plugin: \n from setuptools import setup\n\nVERSION = \"0.1\"\n\nsetup(\n name=\"datasette-plugin-demos\",\n description=\"Examples of plugins for Datasette\",\n author=\"Simon Willison\",\n url=\"https://github.com/simonw/datasette-plugin-demos\",\n license=\"Apache License, Version 2.0\",\n version=VERSION,\n py_modules=[\"datasette_plugin_demos\"],\n entry_points={\n \"datasette\": [\n \"plugin_demos = datasette_plugin_demos\"\n ]\n },\n install_requires=[\"datasette\"],\n) \n And a Python module file, datasette_plugin_demos.py , that implements the plugin: \n from datasette import hookimpl\nimport random\n\n\n@hookimpl\ndef prepare_jinja2_environment(env):\n env.filters[\"uppercase\"] = lambda u: u.upper()\n\n\n@hookimpl\ndef prepare_connection(conn):\n conn.create_function(\n \"random_integer\", 2, random.randint\n ) \n Having built a plugin in this way you can turn it into an installable package using the following command: \n python3 setup.py sdist \n This will create a .tar.gz file in the dist/ directory. \n You can then install your new plugin into a Datasette virtual environment or Docker container using pip : \n pip install datasette-plugin-demos-0.1.tar.gz \n To learn how to upload your plugin to PyPI for use by other people, read the PyPA guide to Packaging and distributing projects .", "sections_fts": 186, "rank": null} {"rowid": 5, "title": "Static assets", "content": "If your plugin has a static/ directory, Datasette will automatically configure itself to serve those static assets from the following path: \n /-/static-plugins/NAME_OF_PLUGIN_PACKAGE/yourfile.js \n Use the datasette.urls.static_plugins(plugin_name, path) method to generate URLs to that asset that take the base_url setting into account, see datasette.urls . \n To bundle the static assets for a plugin in the package that you publish to PyPI, add the following to the plugin's setup.py : \n package_data = (\n {\n \"datasette_plugin_name\": [\n \"static/plugin.js\",\n ],\n },\n) \n Where datasette_plugin_name is the name of the plugin package (note that it uses underscores, not hyphens) and static/plugin.js is the path within that package to the static file. \n datasette-cluster-map is a useful example of a plugin that includes packaged static assets in this way.", "sections_fts": 186, "rank": null} {"rowid": 6, "title": "Custom templates", "content": "If your plugin has a templates/ directory, Datasette will attempt to load templates from that directory before it uses its own default templates. \n The priority order for template loading is: \n \n \n templates from the --template-dir argument, if specified \n \n \n templates from the templates/ directory in any installed plugins \n \n \n default templates that ship with Datasette \n \n \n See Custom pages and templates for more details on how to write custom templates, including which filenames to use to customize which parts of the Datasette UI. \n Templates should be bundled for distribution using the same package_data mechanism in setup.py described for static assets above, for example: \n package_data = (\n {\n \"datasette_plugin_name\": [\n \"templates/my_template.html\",\n ],\n },\n) \n You can also use wildcards here such as templates/*.html . See datasette-edit-schema for an example of this pattern.", "sections_fts": 186, "rank": null} {"rowid": 7, "title": "Writing plugins that accept configuration", "content": "When you are writing plugins, you can access plugin configuration like this using the datasette plugin_config() method. If you know you need plugin configuration for a specific table, you can access it like this: \n plugin_config = datasette.plugin_config(\n \"datasette-cluster-map\", database=\"sf-trees\", table=\"Street_Tree_List\"\n) \n This will return the {\"latitude_column\": \"lat\", \"longitude_column\": \"lng\"} in the above example. \n If there is no configuration for that plugin, the method will return None . \n If it cannot find the requested configuration at the table layer, it will fall back to the database layer and then the root layer. For example, a user may have set the plugin configuration option like so: \n {\n \"databases: {\n \"sf-trees\": {\n \"plugins\": {\n \"datasette-cluster-map\": {\n \"latitude_column\": \"xlat\",\n \"longitude_column\": \"xlng\"\n }\n }\n }\n }\n} \n In this case, the above code would return that configuration for ANY table within the sf-trees database. \n The plugin configuration could also be set at the top level of metadata.json : \n {\n \"title\": \"This is the top-level title in metadata.json\",\n \"plugins\": {\n \"datasette-cluster-map\": {\n \"latitude_column\": \"xlat\",\n \"longitude_column\": \"xlng\"\n }\n }\n} \n Now that datasette-cluster-map plugin configuration will apply to every table in every database.", "sections_fts": 186, "rank": null} {"rowid": 8, "title": "Designing URLs for your plugin", "content": "You can register new URL routes within Datasette using the register_routes(datasette) plugin hook. \n Datasette's default URLs include these: \n \n \n /dbname - database page \n \n \n /dbname/tablename - table page \n \n \n /dbname/tablename/pk - row page \n \n \n See Pages and API endpoints and Introspection for more default URL routes. \n To avoid accidentally conflicting with a database file that may be loaded into Datasette, plugins should register URLs using a /-/ prefix. For example, if your plugin adds a new interface for uploading Excel files you might register a URL route like this one: \n \n \n /-/upload-excel \n \n \n Try to avoid registering URLs that clash with other plugins that your users might have installed. There is no central repository of reserved URL paths (yet) but you can review existing plugins by browsing the plugins directory . \n If your plugin includes functionality that relates to a specific database you could also register a URL route like this: \n \n \n /dbname/-/upload-excel \n \n \n Or for a specific table like this: \n \n \n /dbname/tablename/-/modify-table-schema \n \n \n Note that a row could have a primary key of - and this URL scheme will still work, because Datasette row pages do not ever have a trailing slash followed by additional path components.", "sections_fts": 186, "rank": null} {"rowid": 9, "title": "Building URLs within plugins", "content": "Plugins that define their own custom user interface elements may need to link to other pages within Datasette. \n This can be a bit tricky if the Datasette instance is using the base_url configuration setting to run behind a proxy, since that can cause Datasette's URLs to include an additional prefix. \n The datasette.urls object provides internal methods for correctly generating URLs to different pages within Datasette, taking any base_url configuration into account. \n This object is exposed in templates as the urls variable, which can be used like this: \n Back to the Homepage \n See datasette.urls for full details on this object.", "sections_fts": 186, "rank": null} {"rowid": 10, "title": "Facets", "content": "Datasette facets can be used to add a faceted browse interface to any database table.\n With facets, tables are displayed along with a summary showing the most common values in specified columns.\n These values can be selected to further filter the table. \n Here's an example : \n \n Facets can be specified in two ways: using query string parameters, or in metadata.json configuration for the table.", "sections_fts": 186, "rank": null} {"rowid": 11, "title": "Facets in query strings", "content": "To turn on faceting for specific columns on a Datasette table view, add one or more _facet=COLUMN parameters to the URL.\n For example, if you want to turn on facets for the city_id and state columns, construct a URL that looks like this: \n /dbname/tablename?_facet=state&_facet=city_id \n This works for both the HTML interface and the .json view.\n When enabled, facets will cause a facet_results block to be added to the JSON output, looking something like this: \n {\n \"state\": {\n \"name\": \"state\",\n \"results\": [\n {\n \"value\": \"CA\",\n \"label\": \"CA\",\n \"count\": 10,\n \"toggle_url\": \"http://...?_facet=city_id&_facet=state&state=CA\",\n \"selected\": false\n },\n {\n \"value\": \"MI\",\n \"label\": \"MI\",\n \"count\": 4,\n \"toggle_url\": \"http://...?_facet=city_id&_facet=state&state=MI\",\n \"selected\": false\n },\n {\n \"value\": \"MC\",\n \"label\": \"MC\",\n \"count\": 1,\n \"toggle_url\": \"http://...?_facet=city_id&_facet=state&state=MC\",\n \"selected\": false\n }\n ],\n \"truncated\": false\n }\n \"city_id\": {\n \"name\": \"city_id\",\n \"results\": [\n {\n \"value\": 1,\n \"label\": \"San Francisco\",\n \"count\": 6,\n \"toggle_url\": \"http://...?_facet=city_id&_facet=state&city_id=1\",\n \"selected\": false\n },\n {\n \"value\": 2,\n \"label\": \"Los Angeles\",\n \"count\": 4,\n \"toggle_url\": \"http://...?_facet=city_id&_facet=state&city_id=2\",\n \"selected\": false\n },\n {\n \"value\": 3,\n \"label\": \"Detroit\",\n \"count\": 4,\n \"toggle_url\": \"http://...?_facet=city_id&_facet=state&city_id=3\",\n \"selected\": false\n },\n {\n \"value\": 4,\n \"label\": \"Memnonia\",\n \"count\": 1,\n \"toggle_url\": \"http://...?_facet=city_id&_facet=state&city_id=4\",\n \"selected\": false\n }\n ],\n \"truncated\": false\n }\n} \n If Datasette detects that a column is a foreign key, the \"label\" property will be automatically derived from the detected label column on the referenced table. \n The default number of facet results returned is 30, controlled by the default_facet_size setting.\n You can increase this on an individual page by adding ?_facet_size=100 to the query string, up to a maximum of max_returned_rows (which defaults to 1000).", "sections_fts": 186, "rank": null} {"rowid": 12, "title": "Facets in metadata.json", "content": "You can turn facets on by default for specific tables by adding them to a \"facets\" key in a Datasette Metadata file. \n Here's an example that turns on faceting by default for the qLegalStatus column in the Street_Tree_List table in the sf-trees database: \n {\n \"databases\": {\n \"sf-trees\": {\n \"tables\": {\n \"Street_Tree_List\": {\n \"facets\": [\"qLegalStatus\"]\n }\n }\n }\n }\n} \n Facets defined in this way will always be shown in the interface and returned in the API, regardless of the _facet arguments passed to the view. \n You can specify array or date facets in metadata using JSON objects with a single key of array or date and a value specifying the column, like this: \n {\n \"facets\": [\n {\"array\": \"tags\"},\n {\"date\": \"created\"}\n ]\n} \n You can change the default facet size (the number of results shown for each facet) for a table using facet_size : \n {\n \"databases\": {\n \"sf-trees\": {\n \"tables\": {\n \"Street_Tree_List\": {\n \"facets\": [\"qLegalStatus\"],\n \"facet_size\": 10\n }\n }\n }\n }\n}", "sections_fts": 186, "rank": null} {"rowid": 13, "title": "Suggested facets", "content": "Datasette's table UI will suggest facets for the user to apply, based on the following criteria: \n For the currently filtered data are there any columns which, if applied as a facet... \n \n \n Will return 30 or less unique options \n \n \n Will return more than one unique option \n \n \n Will return less unique options than the total number of filtered rows \n \n \n And the query used to evaluate this criteria can be completed in under 50ms \n \n \n That last point is particularly important: Datasette runs a query for every column that is displayed on a page, which could get expensive - so to avoid slow load times it sets a time limit of just 50ms for each of those queries.\n This means suggested facets are unlikely to appear for tables with millions of records in them.", "sections_fts": 186, "rank": null} {"rowid": 14, "title": "Speeding up facets with indexes", "content": "The performance of facets can be greatly improved by adding indexes on the columns you wish to facet by.\n Adding indexes can be performed using the sqlite3 command-line utility. Here's how to add an index on the state column in a table called Food_Trucks : \n $ sqlite3 mydatabase.db\nSQLite version 3.19.3 2017-06-27 16:48:08\nEnter \".help\" for usage hints.\nsqlite> CREATE INDEX Food_Trucks_state ON Food_Trucks(\"state\"); \n Or using the sqlite-utils command-line utility: \n $ sqlite-utils create-index mydatabase.db Food_Trucks state", "sections_fts": 186, "rank": null} {"rowid": 15, "title": "Facet by JSON array", "content": "If your SQLite installation provides the json1 extension (you can check using /-/versions ) Datasette will automatically detect columns that contain JSON arrays of values and offer a faceting interface against those columns. \n This is useful for modelling things like tags without needing to break them out into a new table. \n Example here: latest.datasette.io/fixtures/facetable?_facet_array=tags", "sections_fts": 186, "rank": null} {"rowid": 16, "title": "Facet by date", "content": "If Datasette finds any columns that contain dates in the first 100 values, it will offer a faceting interface against the dates of those values.\n This works especially well against timestamp values such as 2019-03-01 12:44:00 . \n Example here: latest.datasette.io/fixtures/facetable?_facet_date=created", "sections_fts": 186, "rank": null} {"rowid": 17, "title": "SpatiaLite", "content": "The SpatiaLite module for SQLite adds features for handling geographic and spatial data. For an example of what you can do with it, see the tutorial Building a location to time zone API with SpatiaLite . \n To use it with Datasette, you need to install the mod_spatialite dynamic library. This can then be loaded into Datasette using the --load-extension command-line option. \n Datasette can look for SpatiaLite in common installation locations if you run it like this: \n datasette --load-extension=spatialite --setting default_allow_sql off \n If SpatiaLite is in another location, use the full path to the extension instead: \n datasette --setting default_allow_sql off \\\n --load-extension=/usr/local/lib/mod_spatialite.dylib", "sections_fts": 186, "rank": null} {"rowid": 18, "title": "Warning", "content": "The SpatiaLite extension adds a large number of additional SQL functions , some of which are not be safe for untrusted users to execute: they may cause the Datasette server to crash. \n You should not expose a SpatiaLite-enabled Datasette instance to the public internet without taking extra measures to secure it against potentially harmful SQL queries. \n The following steps are recommended: \n \n \n Disable arbitrary SQL queries by untrusted users. See Controlling the ability to execute arbitrary SQL for ways to do this. The easiest is to start Datasette with the datasette --setting default_allow_sql off option. \n \n \n Define Canned queries with the SQL queries that use SpatiaLite functions that you want people to be able to execute. \n \n \n The Datasette SpatiaLite tutorial includes detailed instructions for running SpatiaLite safely using these techniques", "sections_fts": 186, "rank": null} {"rowid": 19, "title": "Installation", "content": "", "sections_fts": 186, "rank": null} {"rowid": 20, "title": "Installing SpatiaLite on OS X", "content": "The easiest way to install SpatiaLite on OS X is to use Homebrew . \n brew update\nbrew install spatialite-tools \n This will install the spatialite command-line tool and the mod_spatialite dynamic library. \n You can now run Datasette like so: \n datasette --load-extension=spatialite", "sections_fts": 186, "rank": null} {"rowid": 21, "title": "Installing SpatiaLite on Linux", "content": "SpatiaLite is packaged for most Linux distributions. \n apt install spatialite-bin libsqlite3-mod-spatialite \n Depending on your distribution, you should be able to run Datasette something like this: \n datasette --load-extension=/usr/lib/x86_64-linux-gnu/mod_spatialite.so \n If you are unsure of the location of the module, try running locate mod_spatialite and see what comes back.", "sections_fts": 186, "rank": null} {"rowid": 22, "title": "Spatial indexing latitude/longitude columns", "content": "Here's a recipe for taking a table with existing latitude and longitude columns, adding a SpatiaLite POINT geometry column to that table, populating the new column and then populating a spatial index: \n import sqlite3\n\nconn = sqlite3.connect(\"museums.db\")\n# Lead the spatialite extension:\nconn.enable_load_extension(True)\nconn.load_extension(\"/usr/local/lib/mod_spatialite.dylib\")\n# Initialize spatial metadata for this database:\nconn.execute(\"select InitSpatialMetadata(1)\")\n# Add a geometry column called point_geom to our museums table:\nconn.execute(\n \"SELECT AddGeometryColumn('museums', 'point_geom', 4326, 'POINT', 2);\"\n)\n# Now update that geometry column with the lat/lon points\nconn.execute(\n \"\"\"\n UPDATE museums SET\n point_geom = GeomFromText('POINT('||\"longitude\"||' '||\"latitude\"||')',4326);\n\"\"\"\n)\n# Now add a spatial index to that column\nconn.execute(\n 'select CreateSpatialIndex(\"museums\", \"point_geom\");'\n)\n# If you don't commit your changes will not be persisted:\nconn.commit()\nconn.close()", "sections_fts": 186, "rank": null} {"rowid": 23, "title": "Making use of a spatial index", "content": "SpatiaLite spatial indexes are R*Trees. They allow you to run efficient bounding box queries using a sub-select, with a similar pattern to that used for Searches using custom SQL . \n In the above example, the resulting index will be called idx_museums_point_geom . This takes the form of a SQLite virtual table. You can inspect its contents using the following query: \n select * from idx_museums_point_geom limit 10; \n Here's a live example: timezones-api.datasette.io/timezones/idx_timezones_Geometry \n \n \n \n \n \n \n \n \n \n \n pkid \n \n \n xmin \n \n \n xmax \n \n \n ymin \n \n \n ymax \n \n \n \n \n \n \n 1 \n \n \n -8.601725578308105 \n \n \n -2.4930307865142822 \n \n \n 4.162120819091797 \n \n \n 10.74019718170166 \n \n \n \n \n 2 \n \n \n -3.2607860565185547 \n \n \n 1.27329421043396 \n \n \n 4.539252281188965 \n \n \n 11.174856185913086 \n \n \n \n \n 3 \n \n \n 32.997581481933594 \n \n \n 47.98238754272461 \n \n \n 3.3974475860595703 \n \n \n 14.894054412841797 \n \n \n \n \n 4 \n \n \n -8.66890811920166 \n \n \n 11.997337341308594 \n \n \n 18.9681453704834 \n \n \n 37.296207427978516 \n \n \n \n \n 5 \n \n \n 36.43336486816406 \n \n \n 43.300174713134766 \n \n \n 12.354820251464844 \n \n \n 18.070993423461914 \n \n \n \n \n \n You can now construct efficient bounding box queries that will make use of the index like this: \n select * from museums where museums.rowid in (\n SELECT pkid FROM idx_museums_point_geom\n -- left-hand-edge of point > left-hand-edge of bbox (minx)\n where xmin > :bbox_minx\n -- right-hand-edge of point < right-hand-edge of bbox (maxx)\n and xmax < :bbox_maxx\n -- bottom-edge of point > bottom-edge of bbox (miny)\n and ymin > :bbox_miny\n -- top-edge of point < top-edge of bbox (maxy)\n and ymax < :bbox_maxy\n); \n Spatial indexes can be created against polygon columns as well as point columns, in which case they will represent the minimum bounding rectangle of that polygon. This is useful for accelerating within queries, as seen in the Timezones API example.", "sections_fts": 186, "rank": null} {"rowid": 24, "title": "Importing shapefiles into SpatiaLite", "content": "The shapefile format is a common format for distributing geospatial data. You can use the spatialite command-line tool to create a new database table from a shapefile. \n Try it now with the North America shapefile available from the University of North Carolina Global River Database project. Download the file and unzip it (this will create files called narivs.dbf , narivs.prj , narivs.shp and narivs.shx in the current directory), then run the following: \n $ spatialite rivers-database.db\nSpatiaLite version ..: 4.3.0a Supported Extensions:\n...\nspatialite> .loadshp narivs rivers CP1252 23032\n========\nLoading shapefile at 'narivs' into SQLite table 'rivers'\n...\nInserted 467973 rows into 'rivers' from SHAPEFILE \n This will load the data from the narivs shapefile into a new database table called rivers . \n Exit out of spatialite (using Ctrl+D ) and run Datasette against your new database like this: \n datasette rivers-database.db \\\n --load-extension=/usr/local/lib/mod_spatialite.dylib \n If you browse to http://localhost:8001/rivers-database/rivers you will see the new table... but the Geometry column will contain unreadable binary data (SpatiaLite uses a custom format based on WKB ). \n The easiest way to turn this into semi-readable data is to use the SpatiaLite AsGeoJSON function. Try the following using the SQL query interface at http://localhost:8001/rivers-database : \n select *, AsGeoJSON(Geometry) from rivers limit 10; \n This will give you back an additional column of GeoJSON. You can copy and paste GeoJSON from this column into the debugging tool at geojson.io to visualize it on a map. \n To see a more interesting example, try ordering the records with the longest geometry first. Since there are 467,000 rows in the table you will first need to increase the SQL time limit imposed by Datasette: \n datasette rivers-database.db \\\n --load-extension=/usr/local/lib/mod_spatialite.dylib \\\n --setting sql_time_limit_ms 10000 \n Now try the following query: \n select *, AsGeoJSON(Geometry) from rivers\norder by length(Geometry) desc limit 10;", "sections_fts": 186, "rank": null} {"rowid": 25, "title": "Importing GeoJSON polygons using Shapely", "content": "Another common form of polygon data is the GeoJSON format. This can be imported into SpatiaLite directly, or by using the Shapely Python library. \n Who's On First is an excellent source of openly licensed GeoJSON polygons. Let's import the geographical polygon for Wales. First, we can use the Who's On First Spelunker tool to find the record for Wales: \n spelunker.whosonfirst.org/id/404227475 \n That page includes a link to the GeoJSON record, which can be accessed here: \n data.whosonfirst.org/404/227/475/404227475.geojson \n Here's Python code to create a SQLite database, enable SpatiaLite, create a places table and then add a record for Wales: \n import sqlite3\n\nconn = sqlite3.connect(\"places.db\")\n# Enable SpatialLite extension\nconn.enable_load_extension(True)\nconn.load_extension(\"/usr/local/lib/mod_spatialite.dylib\")\n# Create the masic countries table\nconn.execute(\"select InitSpatialMetadata(1)\")\nconn.execute(\n \"create table places (id integer primary key, name text);\"\n)\n# Add a MULTIPOLYGON Geometry column\nconn.execute(\n \"SELECT AddGeometryColumn('places', 'geom', 4326, 'MULTIPOLYGON', 2);\"\n)\n# Add a spatial index against the new column\nconn.execute(\"SELECT CreateSpatialIndex('places', 'geom');\")\n# Now populate the table\nfrom shapely.geometry.multipolygon import MultiPolygon\nfrom shapely.geometry import shape\nimport requests\n\ngeojson = requests.get(\n \"https://data.whosonfirst.org/404/227/475/404227475.geojson\"\n).json()\n# Convert to \"Well Known Text\" format\nwkt = shape(geojson[\"geometry\"]).wkt\n# Insert and commit the record\nconn.execute(\n \"INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))\",\n (\"Wales\", wkt),\n)\nconn.commit()", "sections_fts": 186, "rank": null} {"rowid": 26, "title": "Querying polygons using within()", "content": "The within() SQL function can be used to check if a point is within a geometry: \n select\n name\nfrom\n places\nwhere\n within(GeomFromText('POINT(-3.1724366 51.4704448)'), places.geom); \n The GeomFromText() function takes a string of well-known text. Note that the order used here is longitude then latitude . \n To run that same within() query in a way that benefits from the spatial index, use the following: \n select\n name\nfrom\n places\nwhere\n within(GeomFromText('POINT(-3.1724366 51.4704448)'), places.geom)\n and rowid in (\n SELECT pkid FROM idx_places_geom\n where xmin < -3.1724366\n and xmax > -3.1724366\n and ymin < 51.4704448\n and ymax > 51.4704448\n );", "sections_fts": 186, "rank": null} {"rowid": 27, "title": "Plugin hooks", "content": "Datasette plugins use plugin hooks to customize Datasette's behavior. These hooks are powered by the pluggy plugin system. \n Each plugin can implement one or more hooks using the @hookimpl decorator against a function named that matches one of the hooks documented on this page. \n When you implement a plugin hook you can accept any or all of the parameters that are documented as being passed to that hook. \n For example, you can implement the render_cell plugin hook like this even though the full documented hook signature is render_cell(row, value, column, table, database, datasette) : \n @hookimpl\ndef render_cell(value, column):\n if column == \"stars\":\n return \"*\" * int(value) \n \n List of plugin hooks \n \n \n prepare_connection(conn, database, datasette) \n \n \n prepare_jinja2_environment(env, datasette) \n \n \n extra_template_vars(template, database, table, columns, view_name, request, datasette) \n \n \n extra_css_urls(template, database, table, columns, view_name, request, datasette) \n \n \n extra_js_urls(template, database, table, columns, view_name, request, datasette) \n \n \n extra_body_script(template, database, table, columns, view_name, request, datasette) \n \n \n publish_subcommand(publish) \n \n \n render_cell(row, value, column, table, database, datasette) \n \n \n register_output_renderer(datasette) \n \n \n register_routes(datasette) \n \n \n register_commands(cli) \n \n \n register_facet_classes() \n \n \n asgi_wrapper(datasette) \n \n \n startup(datasette) \n \n \n canned_queries(datasette, database, actor) \n \n \n actor_from_request(datasette, request) \n \n \n filters_from_request(request, database, table, datasette) \n \n \n permission_allowed(datasette, actor, action, resource) \n \n \n register_magic_parameters(datasette) \n \n \n forbidden(datasette, request, message) \n \n \n handle_exception(datasette, request, exception) \n \n \n menu_links(datasette, actor, request) \n \n \n table_actions(datasette, actor, database, table, request) \n \n \n database_actions(datasette, actor, database, request) \n \n \n skip_csrf(datasette, scope) \n \n \n get_metadata(datasette, key, database, table)", "sections_fts": 186, "rank": null} {"rowid": 28, "title": "prepare_connection(conn, database, datasette)", "content": "conn - sqlite3 connection object \n \n The connection that is being opened \n \n \n \n database - string \n \n The name of the database \n \n \n \n datasette - Datasette class \n \n You can use this to access plugin configuration options via datasette.plugin_config(your_plugin_name) \n \n \n \n This hook is called when a new SQLite database connection is created. You can\n use it to register custom SQL functions ,\n aggregates and collations. For example: \n from datasette import hookimpl\nimport random\n\n\n@hookimpl\ndef prepare_connection(conn):\n conn.create_function(\n \"random_integer\", 2, random.randint\n ) \n This registers a SQL function called random_integer which takes two\n arguments and can be called like this: \n select random_integer(1, 10); \n Examples: datasette-jellyfish , datasette-jq , datasette-haversine , datasette-rure", "sections_fts": 186, "rank": null} {"rowid": 29, "title": "prepare_jinja2_environment(env, datasette)", "content": "env - jinja2 Environment \n \n The template environment that is being prepared \n \n \n \n datasette - Datasette class \n \n You can use this to access plugin configuration options via datasette.plugin_config(your_plugin_name) \n \n \n \n This hook is called with the Jinja2 environment that is used to evaluate\n Datasette HTML templates. You can use it to do things like register custom\n template filters , for\n example: \n from datasette import hookimpl\n\n\n@hookimpl\ndef prepare_jinja2_environment(env):\n env.filters[\"uppercase\"] = lambda u: u.upper() \n You can now use this filter in your custom templates like so: \n Table name: {{ table|uppercase }} \n This function can return an awaitable function if it needs to run any async code. \n Examples: datasette-edit-templates", "sections_fts": 186, "rank": null} {"rowid": 30, "title": "extra_template_vars(template, database, table, columns, view_name, request, datasette)", "content": "Extra template variables that should be made available in the rendered template context. \n \n \n template - string \n \n The template that is being rendered, e.g. database.html \n \n \n \n database - string or None \n \n The name of the database, or None if the page does not correspond to a database (e.g. the root page) \n \n \n \n table - string or None \n \n The name of the table, or None if the page does not correct to a table \n \n \n \n columns - list of strings or None \n \n The names of the database columns that will be displayed on this page. None if the page does not contain a table. \n \n \n \n view_name - string \n \n The name of the view being displayed. ( index , database , table , and row are the most important ones.) \n \n \n \n request - Request object or None \n \n The current HTTP request. This can be None if the request object is not available. \n \n \n \n datasette - Datasette class \n \n You can use this to access plugin configuration options via datasette.plugin_config(your_plugin_name) \n \n \n \n This hook can return one of three different types: \n \n \n Dictionary \n \n If you return a dictionary its keys and values will be merged into the template context. \n \n \n \n Function that returns a dictionary \n \n If you return a function it will be executed. If it returns a dictionary those values will will be merged into the template context. \n \n \n \n Function that returns an awaitable function that returns a dictionary \n \n You can also return a function which returns an awaitable function which returns a dictionary. \n \n \n \n Datasette runs Jinja2 in async mode , which means you can add awaitable functions to the template scope and they will be automatically awaited when they are rendered by the template. \n Here's an example plugin that adds a \"user_agent\" variable to the template context containing the current request's User-Agent header: \n @hookimpl\ndef extra_template_vars(request):\n return {\"user_agent\": request.headers.get(\"user-agent\")} \n This example returns an awaitable function which adds a list of hidden_table_names to the context: \n @hookimpl\ndef extra_template_vars(datasette, database):\n async def hidden_table_names():\n if database:\n db = datasette.databases[database]\n return {\n \"hidden_table_names\": await db.hidden_table_names()\n }\n else:\n return {}\n\n return hidden_table_names \n And here's an example which adds a sql_first(sql_query) function which executes a SQL statement and returns the first column of the first row of results: \n @hookimpl\ndef extra_template_vars(datasette, database):\n async def sql_first(sql, dbname=None):\n dbname = (\n dbname\n or database\n or next(iter(datasette.databases.keys()))\n )\n result = await datasette.execute(dbname, sql)\n return result.rows[0][0]\n\n return {\"sql_first\": sql_first} \n You can then use the new function in a template like so: \n SQLite version: {{ sql_first(\"select sqlite_version()\") }} \n Examples: datasette-search-all , datasette-template-sql", "sections_fts": 186, "rank": null} {"rowid": 31, "title": "extra_css_urls(template, database, table, columns, view_name, request, datasette)", "content": "This takes the same arguments as extra_template_vars(...) \n Return a list of extra CSS URLs that should be included on the page. These can\n take advantage of the CSS class hooks described in Custom pages and templates . \n This can be a list of URLs: \n from datasette import hookimpl\n\n\n@hookimpl\ndef extra_css_urls():\n return [\n \"https://stackpath.bootstrapcdn.com/bootstrap/4.1.0/css/bootstrap.min.css\"\n ] \n Or a list of dictionaries defining both a URL and an\n SRI hash : \n @hookimpl\ndef extra_css_urls():\n return [\n {\n \"url\": \"https://stackpath.bootstrapcdn.com/bootstrap/4.1.0/css/bootstrap.min.css\",\n \"sri\": \"sha384-9gVQ4dYFwwWSjIDZnLEWnxCjeSWFphJiwGPXr1jddIhOegiu1FwO5qRGvFXOdJZ4\",\n }\n ] \n This function can also return an awaitable function, useful if it needs to run any async code: \n @hookimpl\ndef extra_css_urls(datasette):\n async def inner():\n db = datasette.get_database()\n results = await db.execute(\n \"select url from css_files\"\n )\n return [r[0] for r in results]\n\n return inner \n Examples: datasette-cluster-map , datasette-vega", "sections_fts": 186, "rank": null} {"rowid": 32, "title": "extra_js_urls(template, database, table, columns, view_name, request, datasette)", "content": "This takes the same arguments as extra_template_vars(...) \n This works in the same way as extra_css_urls() but for JavaScript. You can\n return a list of URLs, a list of dictionaries or an awaitable function that returns those things: \n from datasette import hookimpl\n\n\n@hookimpl\ndef extra_js_urls():\n return [\n {\n \"url\": \"https://code.jquery.com/jquery-3.3.1.slim.min.js\",\n \"sri\": \"sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo\",\n }\n ] \n You can also return URLs to files from your plugin's static/ directory, if\n you have one: \n @hookimpl\ndef extra_js_urls():\n return [\"/-/static-plugins/your-plugin/app.js\"] \n Note that your-plugin here should be the hyphenated plugin name - the name that is displayed in the list on the /-/plugins debug page. \n If your code uses JavaScript modules you should include the \"module\": True key. See Custom CSS and JavaScript for more details. \n @hookimpl\ndef extra_js_urls():\n return [\n {\n \"url\": \"/-/static-plugins/your-plugin/app.js\",\n \"module\": True,\n }\n ] \n Examples: datasette-cluster-map , datasette-vega", "sections_fts": 186, "rank": null} {"rowid": 33, "title": "extra_body_script(template, database, table, columns, view_name, request, datasette)", "content": "Extra JavaScript to be added to a element: \n @hookimpl\ndef extra_body_script():\n return {\n \"module\": True,\n \"script\": \"console.log('Your JavaScript goes here...')\",\n } \n This will add the following to the end of your page: \n \n Example: datasette-cluster-map", "sections_fts": 186, "rank": null} {"rowid": 34, "title": "publish_subcommand(publish)", "content": "publish - Click publish command group \n \n The Click command group for the datasette publish subcommand \n \n \n \n This hook allows you to create new providers for the datasette publish \n command. Datasette uses this hook internally to implement the default cloudrun \n and heroku subcommands, so you can read\n their source \n to see examples of this hook in action. \n Let's say you want to build a plugin that adds a datasette publish my_hosting_provider --api_key=xxx mydatabase.db publish command. Your implementation would start like this: \n from datasette import hookimpl\nfrom datasette.publish.common import (\n add_common_publish_arguments_and_options,\n)\nimport click\n\n\n@hookimpl\ndef publish_subcommand(publish):\n @publish.command()\n @add_common_publish_arguments_and_options\n @click.option(\n \"-k\",\n \"--api_key\",\n help=\"API key for talking to my hosting provider\",\n )\n def my_hosting_provider(\n files,\n metadata,\n extra_options,\n branch,\n template_dir,\n plugins_dir,\n static,\n install,\n plugin_secret,\n version_note,\n secret,\n title,\n license,\n license_url,\n source,\n source_url,\n about,\n about_url,\n api_key,\n ):\n ... \n Examples: datasette-publish-fly , datasette-publish-vercel", "sections_fts": 186, "rank": null} {"rowid": 35, "title": "render_cell(row, value, column, table, database, datasette)", "content": "Lets you customize the display of values within table cells in the HTML table view. \n \n \n row - sqlite.Row \n \n The SQLite row object that the value being rendered is part of \n \n \n \n value - string, integer, float, bytes or None \n \n The value that was loaded from the database \n \n \n \n column - string \n \n The name of the column being rendered \n \n \n \n table - string or None \n \n The name of the table - or None if this is a custom SQL query \n \n \n \n database - string \n \n The name of the database \n \n \n \n datasette - Datasette class \n \n You can use this to access plugin configuration options via datasette.plugin_config(your_plugin_name) , or to execute SQL queries. \n \n \n \n If your hook returns None , it will be ignored. Use this to indicate that your hook is not able to custom render this particular value. \n If the hook returns a string, that string will be rendered in the table cell. \n If you want to return HTML markup you can do so by returning a jinja2.Markup object. \n You can also return an awaitable function which returns a value. \n Datasette will loop through all available render_cell hooks and display the value returned by the first one that does not return None . \n Here is an example of a custom render_cell() plugin which looks for values that are a JSON string matching the following format: \n {\"href\": \"https://www.example.com/\", \"label\": \"Name\"} \n If the value matches that pattern, the plugin returns an HTML link element: \n from datasette import hookimpl\nimport markupsafe\nimport json\n\n\n@hookimpl\ndef render_cell(value):\n # Render {\"href\": \"...\", \"label\": \"...\"} as link\n if not isinstance(value, str):\n return None\n stripped = value.strip()\n if not (\n stripped.startswith(\"{\") and stripped.endswith(\"}\")\n ):\n return None\n try:\n data = json.loads(value)\n except ValueError:\n return None\n if not isinstance(data, dict):\n return None\n if set(data.keys()) != {\"href\", \"label\"}:\n return None\n href = data[\"href\"]\n if not (\n href.startswith(\"/\")\n or href.startswith(\"http://\")\n or href.startswith(\"https://\")\n ):\n return None\n return markupsafe.Markup(\n '{label}'.format(\n href=markupsafe.escape(data[\"href\"]),\n label=markupsafe.escape(data[\"label\"] or \"\")\n or \" \",\n )\n ) \n Examples: datasette-render-binary , datasette-render-markdown , datasette-json-html", "sections_fts": 186, "rank": null} {"rowid": 36, "title": "register_output_renderer(datasette)", "content": "datasette - Datasette class \n \n You can use this to access plugin configuration options via datasette.plugin_config(your_plugin_name) \n \n \n \n Registers a new output renderer, to output data in a custom format. The hook function should return a dictionary, or a list of dictionaries, of the following shape: \n @hookimpl\ndef register_output_renderer(datasette):\n return {\n \"extension\": \"test\",\n \"render\": render_demo,\n \"can_render\": can_render_demo, # Optional\n } \n This will register render_demo to be called when paths with the extension .test (for example /database.test , /database/table.test , or /database/table/row.test ) are requested. \n render_demo is a Python function. It can be a regular function or an async def render_demo() awaitable function, depending on if it needs to make any asynchronous calls. \n can_render_demo is a Python function (or async def function) which accepts the same arguments as render_demo but just returns True or False . It lets Datasette know if the current SQL query can be represented by the plugin - and hence influnce if a link to this output format is displayed in the user interface. If you omit the \"can_render\" key from the dictionary every query will be treated as being supported by the plugin. \n When a request is received, the \"render\" callback function is called with zero or more of the following arguments. Datasette will inspect your callback function and pass arguments that match its function signature. \n \n \n datasette - Datasette class \n \n For accessing plugin configuration and executing queries. \n \n \n \n columns - list of strings \n \n The names of the columns returned by this query. \n \n \n \n rows - list of sqlite3.Row objects \n \n The rows returned by the query. \n \n \n \n sql - string \n \n The SQL query that was executed. \n \n \n \n query_name - string or None \n \n If this was the execution of a canned query , the name of that query. \n \n \n \n database - string \n \n The name of the database. \n \n \n \n table - string or None \n \n The table or view, if one is being rendered. \n \n \n \n request - Request object \n \n The current HTTP request. \n \n \n \n view_name - string \n \n The name of the current view being called. index , database , table , and row are the most important ones. \n \n \n \n The callback function can return None , if it is unable to render the data, or a Response class that will be returned to the caller. \n It can also return a dictionary with the following keys. This format is deprecated as-of Datasette 0.49 and will be removed by Datasette 1.0. \n \n \n body - string or bytes, optional \n \n The response body, default empty \n \n \n \n content_type - string, optional \n \n The Content-Type header, default text/plain \n \n \n \n status_code - integer, optional \n \n The HTTP status code, default 200 \n \n \n \n headers - dictionary, optional \n \n Extra HTTP headers to be returned in the response. \n \n \n \n An example of an output renderer callback function: \n def render_demo():\n return Response.text(\"Hello World\") \n Here is a more complex example: \n async def render_demo(datasette, columns, rows):\n db = datasette.get_database()\n result = await db.execute(\"select sqlite_version()\")\n first_row = \" | \".join(columns)\n lines = [first_row]\n lines.append(\"=\" * len(first_row))\n for row in rows:\n lines.append(\" | \".join(row))\n return Response(\n \"\\n\".join(lines),\n content_type=\"text/plain; charset=utf-8\",\n headers={\"x-sqlite-version\": result.first()[0]},\n ) \n And here is an example can_render function which returns True only if the query results contain the columns atom_id , atom_title and atom_updated : \n def can_render_demo(columns):\n return {\n \"atom_id\",\n \"atom_title\",\n \"atom_updated\",\n }.issubset(columns) \n Examples: datasette-atom , datasette-ics , datasette-geojson , datasette-copyable", "sections_fts": 186, "rank": null} {"rowid": 37, "title": "register_routes(datasette)", "content": "datasette - Datasette class \n \n You can use this to access plugin configuration options via datasette.plugin_config(your_plugin_name) \n \n \n \n Register additional view functions to execute for specified URL routes. \n Return a list of (regex, view_function) pairs, something like this: \n from datasette import hookimpl, Response\nimport html\n\n\nasync def hello_from(request):\n name = request.url_vars[\"name\"]\n return Response.html(\n \"Hello from {}\".format(html.escape(name))\n )\n\n\n@hookimpl\ndef register_routes():\n return [(r\"^/hello-from/(?P.*)$\", hello_from)] \n The view functions can take a number of different optional arguments. The corresponding argument will be passed to your function depending on its named parameters - a form of dependency injection. \n The optional view function arguments are as follows: \n \n \n datasette - Datasette class \n \n You can use this to access plugin configuration options via datasette.plugin_config(your_plugin_name) , or to execute SQL queries. \n \n \n \n request - Request object \n \n The current HTTP request. \n \n \n \n scope - dictionary \n \n The incoming ASGI scope dictionary. \n \n \n \n send - function \n \n The ASGI send function. \n \n \n \n receive - function \n \n The ASGI receive function. \n \n \n \n The view function can be a regular function or an async def function, depending on if it needs to use any await APIs. \n The function can either return a Response class or it can return nothing and instead respond directly to the request using the ASGI send function (for advanced uses only). \n It can also raise the datasette.NotFound exception to return a 404 not found error, or the datasette.Forbidden exception for a 403 forbidden. \n See Designing URLs for your plugin for tips on designing the URL routes used by your plugin. \n Examples: datasette-auth-github , datasette-psutil", "sections_fts": 186, "rank": null} {"rowid": 38, "title": "register_commands(cli)", "content": "cli - the root Datasette Click command group \n \n Use this to register additional CLI commands \n \n \n \n Register additional CLI commands that can be run using datsette yourcommand ... . This provides a mechanism by which plugins can add new CLI commands to Datasette. \n This example registers a new datasette verify file1.db file2.db command that checks if the provided file paths are valid SQLite databases: \n from datasette import hookimpl\nimport click\nimport sqlite3\n\n\n@hookimpl\ndef register_commands(cli):\n @cli.command()\n @click.argument(\n \"files\", type=click.Path(exists=True), nargs=-1\n )\n def verify(files):\n \"Verify that files can be opened by Datasette\"\n for file in files:\n conn = sqlite3.connect(str(file))\n try:\n conn.execute(\"select * from sqlite_master\")\n except sqlite3.DatabaseError:\n raise click.ClickException(\n \"Invalid database: {}\".format(file)\n ) \n The new command can then be executed like so: \n datasette verify fixtures.db \n Help text (from the docstring for the function plus any defined Click arguments or options) will become available using: \n datasette verify --help \n Plugins can register multiple commands by making multiple calls to the @cli.command() decorator. Consult the Click documentation for full details on how to build a CLI command, including how to define arguments and options. \n Note that register_commands() plugins cannot used with the --plugins-dir mechanism - they need to be installed into the same virtual environment as Datasette using pip install . Provided it has a setup.py file (see Packaging a plugin ) you can run pip install directly against the directory in which you are developing your plugin like so: \n pip install -e path/to/my/datasette-plugin \n Examples: datasette-auth-passwords , datasette-verify", "sections_fts": 186, "rank": null} {"rowid": 39, "title": "register_facet_classes()", "content": "Return a list of additional Facet subclasses to be registered. \n \n The design of this plugin hook is unstable and may change. See issue 830 . \n \n Each Facet subclass implements a new type of facet operation. The class should look like this: \n class SpecialFacet(Facet):\n # This key must be unique across all facet classes:\n type = \"special\"\n\n async def suggest(self):\n # Use self.sql and self.params to suggest some facets\n suggested_facets = []\n suggested_facets.append(\n {\n \"name\": column, # Or other unique name\n # Construct the URL that will enable this facet:\n \"toggle_url\": self.ds.absolute_url(\n self.request,\n path_with_added_args(\n self.request, {\"_facet\": column}\n ),\n ),\n }\n )\n return suggested_facets\n\n async def facet_results(self):\n # This should execute the facet operation and return results, again\n # using self.sql and self.params as the starting point\n facet_results = []\n facets_timed_out = []\n facet_size = self.get_facet_size()\n # Do some calculations here...\n for column in columns_selected_for_facet:\n try:\n facet_results_values = []\n # More calculations...\n facet_results_values.append(\n {\n \"value\": value,\n \"label\": label,\n \"count\": count,\n \"toggle_url\": self.ds.absolute_url(\n self.request, toggle_path\n ),\n \"selected\": selected,\n }\n )\n facet_results.append(\n {\n \"name\": column,\n \"results\": facet_results_values,\n \"truncated\": len(facet_rows_results)\n > facet_size,\n }\n )\n except QueryInterrupted:\n facets_timed_out.append(column)\n\n return facet_results, facets_timed_out \n See datasette/facets.py for examples of how these classes can work. \n The plugin hook can then be used to register the new facet class like this: \n @hookimpl\ndef register_facet_classes():\n return [SpecialFacet]", "sections_fts": 186, "rank": null} {"rowid": 40, "title": "asgi_wrapper(datasette)", "content": "Return an ASGI middleware wrapper function that will be applied to the Datasette ASGI application. \n This is a very powerful hook. You can use it to manipulate the entire Datasette response, or even to configure new URL routes that will be handled by your own custom code. \n You can write your ASGI code directly against the low-level specification, or you can use the middleware utilities provided by an ASGI framework such as Starlette . \n This example plugin adds a x-databases HTTP header listing the currently attached databases: \n from datasette import hookimpl\nfrom functools import wraps\n\n\n@hookimpl\ndef asgi_wrapper(datasette):\n def wrap_with_databases_header(app):\n @wraps(app)\n async def add_x_databases_header(\n scope, receive, send\n ):\n async def wrapped_send(event):\n if event[\"type\"] == \"http.response.start\":\n original_headers = (\n event.get(\"headers\") or []\n )\n event = {\n \"type\": event[\"type\"],\n \"status\": event[\"status\"],\n \"headers\": original_headers\n + [\n [\n b\"x-databases\",\n \", \".join(\n datasette.databases.keys()\n ).encode(\"utf-8\"),\n ]\n ],\n }\n await send(event)\n\n await app(scope, receive, wrapped_send)\n\n return add_x_databases_header\n\n return wrap_with_databases_header \n Examples: datasette-cors , datasette-pyinstrument , datasette-total-page-time", "sections_fts": 186, "rank": null} {"rowid": 41, "title": "startup(datasette)", "content": "This hook fires when the Datasette application server first starts up. You can implement a regular function, for example to validate required plugin configuration: \n @hookimpl\ndef startup(datasette):\n config = datasette.plugin_config(\"my-plugin\") or {}\n assert (\n \"required-setting\" in config\n ), \"my-plugin requires setting required-setting\" \n Or you can return an async function which will be awaited on startup. Use this option if you need to make any database queries: \n @hookimpl\ndef startup(datasette):\n async def inner():\n db = datasette.get_database()\n if \"my_table\" not in await db.table_names():\n await db.execute_write(\n \"\"\"\n create table my_table (mycol text)\n \"\"\"\n )\n\n return inner \n Potential use-cases: \n \n \n Run some initialization code for the plugin \n \n \n Create database tables that a plugin needs on startup \n \n \n Validate the metadata configuration for a plugin on startup, and raise an error if it is invalid \n \n \n \n If you are writing unit tests for a plugin that uses this hook and doesn't exercise Datasette by sending\n any simulated requests through it you will need to explicitly call await ds.invoke_startup() in your tests. An example: \n @pytest.mark.asyncio\nasync def test_my_plugin():\n ds = Datasette()\n await ds.invoke_startup()\n # Rest of test goes here \n \n Examples: datasette-saved-queries , datasette-init", "sections_fts": 186, "rank": null} {"rowid": 42, "title": "canned_queries(datasette, database, actor)", "content": "datasette - Datasette class \n \n You can use this to access plugin configuration options via datasette.plugin_config(your_plugin_name) , or to execute SQL queries. \n \n \n \n database - string \n \n The name of the database. \n \n \n \n actor - dictionary or None \n \n The currently authenticated actor . \n \n \n \n Use this hook to return a dictionary of additional canned query definitions for the specified database. The return value should be the same shape as the JSON described in the canned query documentation. \n from datasette import hookimpl\n\n\n@hookimpl\ndef canned_queries(datasette, database):\n if database == \"mydb\":\n return {\n \"my_query\": {\n \"sql\": \"select * from my_table where id > :min_id\"\n }\n } \n The hook can alternatively return an awaitable function that returns a list. Here's an example that returns queries that have been stored in the saved_queries database table, if one exists: \n from datasette import hookimpl\n\n\n@hookimpl\ndef canned_queries(datasette, database):\n async def inner():\n db = datasette.get_database(database)\n if await db.table_exists(\"saved_queries\"):\n results = await db.execute(\n \"select name, sql from saved_queries\"\n )\n return {\n result[\"name\"]: {\"sql\": result[\"sql\"]}\n for result in results\n }\n\n return inner \n The actor parameter can be used to include the currently authenticated actor in your decision. Here's an example that returns saved queries that were saved by that actor: \n from datasette import hookimpl\n\n\n@hookimpl\ndef canned_queries(datasette, database, actor):\n async def inner():\n db = datasette.get_database(database)\n if actor is not None and await db.table_exists(\n \"saved_queries\"\n ):\n results = await db.execute(\n \"select name, sql from saved_queries where actor_id = :id\",\n {\"id\": actor[\"id\"]},\n )\n return {\n result[\"name\"]: {\"sql\": result[\"sql\"]}\n for result in results\n }\n\n return inner \n Example: datasette-saved-queries", "sections_fts": 186, "rank": null} {"rowid": 43, "title": "actor_from_request(datasette, request)", "content": "datasette - Datasette class \n \n You can use this to access plugin configuration options via datasette.plugin_config(your_plugin_name) , or to execute SQL queries. \n \n \n \n request - Request object \n \n The current HTTP request. \n \n \n \n This is part of Datasette's authentication and permissions system . The function should attempt to authenticate an actor (either a user or an API actor of some sort) based on information in the request. \n If it cannot authenticate an actor, it should return None . Otherwise it should return a dictionary representing that actor. \n Here's an example that authenticates the actor based on an incoming API key: \n from datasette import hookimpl\nimport secrets\n\nSECRET_KEY = \"this-is-a-secret\"\n\n\n@hookimpl\ndef actor_from_request(datasette, request):\n authorization = (\n request.headers.get(\"authorization\") or \"\"\n )\n expected = \"Bearer {}\".format(SECRET_KEY)\n\n if secrets.compare_digest(authorization, expected):\n return {\"id\": \"bot\"} \n If you install this in your plugins directory you can test it like this: \n $ curl -H 'Authorization: Bearer this-is-a-secret' http://localhost:8003/-/actor.json \n Instead of returning a dictionary, this function can return an awaitable function which itself returns either None or a dictionary. This is useful for authentication functions that need to make a database query - for example: \n from datasette import hookimpl\n\n\n@hookimpl\ndef actor_from_request(datasette, request):\n async def inner():\n token = request.args.get(\"_token\")\n if not token:\n return None\n # Look up ?_token=xxx in sessions table\n result = await datasette.get_database().execute(\n \"select count(*) from sessions where token = ?\",\n [token],\n )\n if result.first()[0]:\n return {\"token\": token}\n else:\n return None\n\n return inner \n Example: datasette-auth-tokens", "sections_fts": 186, "rank": null} {"rowid": 44, "title": "filters_from_request(request, database, table, datasette)", "content": "request - Request object \n \n The current HTTP request. \n \n \n \n database - string \n \n The name of the database. \n \n \n \n table - string \n \n The name of the table. \n \n \n \n datasette - Datasette class \n \n You can use this to access plugin configuration options via datasette.plugin_config(your_plugin_name) , or to execute SQL queries. \n \n \n \n This hook runs on the table page, and can influence the where clause of the SQL query used to populate that page, based on query string arguments on the incoming request. \n The hook should return an instance of datasette.filters.FilterArguments which has one required and three optional arguments: \n return FilterArguments(\n where_clauses=[\"id > :max_id\"],\n params={\"max_id\": 5},\n human_descriptions=[\"max_id is greater than 5\"],\n extra_context={},\n) \n The arguments to the FilterArguments class constructor are as follows: \n \n \n where_clauses - list of strings, required \n \n A list of SQL fragments that will be inserted into the SQL query, joined by the and operator. These can include :named parameters which will be populated using data in params . \n \n \n \n params - dictionary, optional \n \n Additional keyword arguments to be used when the query is executed. These should match any :arguments in the where clauses. \n \n \n \n human_descriptions - list of strings, optional \n \n These strings will be included in the human-readable description at the top of the page and the page . \n \n \n \n extra_context - dictionary, optional \n \n Additional context variables that should be made available to the table.html template when it is rendered. \n \n \n \n This example plugin causes 0 results to be returned if ?_nothing=1 is added to the URL: \n from datasette import hookimpl\nfrom datasette.filters import FilterArguments\n\n\n@hookimpl\ndef filters_from_request(self, request):\n if request.args.get(\"_nothing\"):\n return FilterArguments(\n [\"1 = 0\"], human_descriptions=[\"NOTHING\"]\n ) \n Example: datasette-leaflet-freedraw", "sections_fts": 186, "rank": null} {"rowid": 45, "title": "permission_allowed(datasette, actor, action, resource)", "content": "datasette - Datasette class \n \n You can use this to access plugin configuration options via datasette.plugin_config(your_plugin_name) , or to execute SQL queries. \n \n \n \n actor - dictionary \n \n The current actor, as decided by actor_from_request(datasette, request) . \n \n \n \n action - string \n \n The action to be performed, e.g. \"edit-table\" . \n \n \n \n resource - string or None \n \n An identifier for the individual resource, e.g. the name of the table. \n \n \n \n Called to check that an actor has permission to perform an action on a resource. Can return True if the action is allowed, False if the action is not allowed or None if the plugin does not have an opinion one way or the other. \n Here's an example plugin which randomly selects if a permission should be allowed or denied, except for view-instance which always uses the default permission scheme instead. \n from datasette import hookimpl\nimport random\n\n\n@hookimpl\ndef permission_allowed(action):\n if action != \"view-instance\":\n # Return True or False at random\n return random.random() > 0.5\n # Returning None falls back to default permissions \n This function can alternatively return an awaitable function which itself returns True , False or None . You can use this option if you need to execute additional database queries using await datasette.execute(...) . \n Here's an example that allows users to view the admin_log table only if their actor id is present in the admin_users table. It aso disallows arbitrary SQL queries for the staff.db database for all users. \n @hookimpl\ndef permission_allowed(datasette, actor, action, resource):\n async def inner():\n if action == \"execute-sql\" and resource == \"staff\":\n return False\n if action == \"view-table\" and resource == (\n \"staff\",\n \"admin_log\",\n ):\n if not actor:\n return False\n user_id = actor[\"id\"]\n return await datasette.get_database(\n \"staff\"\n ).execute(\n \"select count(*) from admin_users where user_id = :user_id\",\n {\"user_id\": user_id},\n )\n\n return inner \n See built-in permissions for a full list of permissions that are included in Datasette core. \n Example: datasette-permissions-sql", "sections_fts": 186, "rank": null} {"rowid": 46, "title": "register_magic_parameters(datasette)", "content": "datasette - Datasette class \n \n You can use this to access plugin configuration options via datasette.plugin_config(your_plugin_name) . \n \n \n \n Magic parameters can be used to add automatic parameters to canned queries . This plugin hook allows additional magic parameters to be defined by plugins. \n Magic parameters all take this format: _prefix_rest_of_parameter . The prefix indicates which magic parameter function should be called - the rest of the parameter is passed as an argument to that function. \n To register a new function, return it as a tuple of (string prefix, function) from this hook. The function you register should take two arguments: key and request , where key is the rest_of_parameter portion of the parameter and request is the current Request object . \n This example registers two new magic parameters: :_request_http_version returning the HTTP version of the current request, and :_uuid_new which returns a new UUID: \n from uuid import uuid4\n\n\ndef uuid(key, request):\n if key == \"new\":\n return str(uuid4())\n else:\n raise KeyError\n\n\ndef request(key, request):\n if key == \"http_version\":\n return request.scope[\"http_version\"]\n else:\n raise KeyError\n\n\n@hookimpl\ndef register_magic_parameters(datasette):\n return [\n (\"request\", request),\n (\"uuid\", uuid),\n ]", "sections_fts": 186, "rank": null} {"rowid": 47, "title": "forbidden(datasette, request, message)", "content": "datasette - Datasette class \n \n You can use this to access plugin configuration options via datasette.plugin_config(your_plugin_name) , or to render templates or execute SQL queries. \n \n \n \n request - Request object \n \n The current HTTP request. \n \n \n \n message - string \n \n A message hinting at why the request was forbidden. \n \n \n \n Plugins can use this to customize how Datasette responds when a 403 Forbidden error occurs - usually because a page failed a permission check, see Permissions . \n If a plugin hook wishes to react to the error, it should return a Response object . \n This example returns a redirect to a /-/login page: \n from datasette import hookimpl\nfrom urllib.parse import urlencode\n\n\n@hookimpl\ndef forbidden(request, message):\n return Response.redirect(\n \"/-/login?=\" + urlencode({\"message\": message})\n ) \n The function can alternatively return an awaitable function if it needs to make any asynchronous method calls. This example renders a template: \n from datasette import hookimpl, Response\n\n\n@hookimpl\ndef forbidden(datasette):\n async def inner():\n return Response.html(\n await datasette.render_template(\n \"render_message.html\", request=request\n )\n )\n\n return inner", "sections_fts": 186, "rank": null} {"rowid": 48, "title": "handle_exception(datasette, request, exception)", "content": "datasette - Datasette class \n \n You can use this to access plugin configuration options via datasette.plugin_config(your_plugin_name) , or to render templates or execute SQL queries. \n \n \n \n request - Request object \n \n The current HTTP request. \n \n \n \n exception - Exception \n \n The exception that was raised. \n \n \n \n This hook is called any time an unexpected exception is raised. You can use it to record the exception. \n If your handler returns a Response object it will be returned to the client in place of the default Datasette error page. \n The handler can return a response directly, or it can return return an awaitable function that returns a response. \n This example logs an error to Sentry and then renders a custom error page: \n from datasette import hookimpl, Response\nimport sentry_sdk\n\n\n@hookimpl\ndef handle_exception(datasette, exception):\n sentry_sdk.capture_exception(exception)\n\n async def inner():\n return Response.html(\n await datasette.render_template(\n \"custom_error.html\", request=request\n )\n )\n\n return inner \n Example: datasette-sentry", "sections_fts": 186, "rank": null} {"rowid": 49, "title": "menu_links(datasette, actor, request)", "content": "datasette - Datasette class \n \n You can use this to access plugin configuration options via datasette.plugin_config(your_plugin_name) , or to execute SQL queries. \n \n \n \n actor - dictionary or None \n \n The currently authenticated actor . \n \n \n \n request - Request object or None \n \n The current HTTP request. This can be None if the request object is not available. \n \n \n \n This hook allows additional items to be included in the menu displayed by Datasette's top right menu icon. \n The hook should return a list of {\"href\": \"...\", \"label\": \"...\"} menu items. These will be added to the menu. \n It can alternatively return an async def awaitable function which returns a list of menu items. \n This example adds a new menu item but only if the signed in user is \"root\" : \n from datasette import hookimpl\n\n\n@hookimpl\ndef menu_links(datasette, actor):\n if actor and actor.get(\"id\") == \"root\":\n return [\n {\n \"href\": datasette.urls.path(\n \"/-/edit-schema\"\n ),\n \"label\": \"Edit schema\",\n },\n ] \n Using datasette.urls here ensures that links in the menu will take the base_url setting into account. \n Examples: datasette-search-all , datasette-graphql", "sections_fts": 186, "rank": null} {"rowid": 50, "title": "table_actions(datasette, actor, database, table, request)", "content": "datasette - Datasette class \n \n You can use this to access plugin configuration options via datasette.plugin_config(your_plugin_name) , or to execute SQL queries. \n \n \n \n actor - dictionary or None \n \n The currently authenticated actor . \n \n \n \n database - string \n \n The name of the database. \n \n \n \n table - string \n \n The name of the table. \n \n \n \n request - Request object or None \n \n The current HTTP request. This can be None if the request object is not available. \n \n \n \n This hook allows table actions to be displayed in a menu accessed via an action icon at the top of the table page. It should return a list of {\"href\": \"...\", \"label\": \"...\"} menu items. \n It can alternatively return an async def awaitable function which returns a list of menu items. \n This example adds a new table action if the signed in user is \"root\" : \n from datasette import hookimpl\n\n\n@hookimpl\ndef table_actions(datasette, actor, database, table):\n if actor and actor.get(\"id\") == \"root\":\n return [\n {\n \"href\": datasette.urls.path(\n \"/-/edit-schema/{}/{}\".format(\n database, table\n )\n ),\n \"label\": \"Edit schema for this table\",\n }\n ] \n Example: datasette-graphql", "sections_fts": 186, "rank": null} {"rowid": 51, "title": "database_actions(datasette, actor, database, request)", "content": "datasette - Datasette class \n \n You can use this to access plugin configuration options via datasette.plugin_config(your_plugin_name) , or to execute SQL queries. \n \n \n \n actor - dictionary or None \n \n The currently authenticated actor . \n \n \n \n database - string \n \n The name of the database. \n \n \n \n request - Request object \n \n The current HTTP request. \n \n \n \n This hook is similar to table_actions(datasette, actor, database, table, request) but populates an actions menu on the database page. \n Example: datasette-graphql", "sections_fts": 186, "rank": null} {"rowid": 52, "title": "skip_csrf(datasette, scope)", "content": "datasette - Datasette class \n \n You can use this to access plugin configuration options via datasette.plugin_config(your_plugin_name) , or to execute SQL queries. \n \n \n \n scope - dictionary \n \n The ASGI scope for the incoming HTTP request. \n \n \n \n This hook can be used to skip CSRF protection for a specific incoming request. For example, you might have a custom path at /submit-comment which is designed to accept comments from anywhere, whether or not the incoming request originated on the site and has an accompanying CSRF token. \n This example will disable CSRF protection for that specific URL path: \n from datasette import hookimpl\n\n\n@hookimpl\ndef skip_csrf(scope):\n return scope[\"path\"] == \"/submit-comment\" \n If any of the currently active skip_csrf() plugin hooks return True , CSRF protection will be skipped for the request.", "sections_fts": 186, "rank": null} {"rowid": 53, "title": "get_metadata(datasette, key, database, table)", "content": "datasette - Datasette class \n \n You can use this to access plugin configuration options via datasette.plugin_config(your_plugin_name) . \n \n \n \n actor - dictionary or None \n \n The currently authenticated actor . \n \n \n \n database - string or None \n \n The name of the database metadata is being asked for. \n \n \n \n table - string or None \n \n The name of the table. \n \n \n \n key - string or None \n \n The name of the key for which data is being asked for. \n \n \n \n This hook is responsible for returning a dictionary corresponding to Datasette Metadata . This function is passed the database , table and key which were passed to the upstream internal request for metadata. Regardless, it is important to return a global metadata object, where \"databases\": [] would be a top-level key. The dictionary returned here, will be merged with, and overwritten by, the contents of the physical metadata.yaml if one is present. \n \n The design of this plugin hook does not currently provide a mechanism for interacting with async code, and may change in the future. See issue 1384 . \n \n @hookimpl\ndef get_metadata(datasette, key, database, table):\n metadata = {\n \"title\": \"This will be the Datasette landing page title!\",\n \"description\": get_instance_description(datasette),\n \"databases\": [],\n }\n for db_name, db_data_dict in get_my_database_meta(\n datasette, database, table, key\n ):\n metadata[\"databases\"][db_name] = db_data_dict\n # whatever we return here will be merged with any other plugins using this hook and\n # will be overwritten by a local metadata.yaml if one exists!\n return metadata \n Example: datasette-remote-metadata plugin", "sections_fts": 186, "rank": null} {"rowid": 54, "title": "Testing plugins", "content": "We recommend using pytest to write automated tests for your plugins. \n If you use the template described in Starting an installable plugin using cookiecutter your plugin will start with a single test in your tests/ directory that looks like this: \n from datasette.app import Datasette\nimport pytest\n\n\n@pytest.mark.asyncio\nasync def test_plugin_is_installed():\n datasette = Datasette(memory=True)\n response = await datasette.client.get(\"/-/plugins.json\")\n assert response.status_code == 200\n installed_plugins = {p[\"name\"] for p in response.json()}\n assert (\n \"datasette-plugin-template-demo\"\n in installed_plugins\n ) \n This test uses the datasette.client object to exercise a test instance of Datasette. datasette.client is a wrapper around the HTTPX Python library which can imitate HTTP requests using ASGI. This is the recommended way to write tests against a Datasette instance. \n This test also uses the pytest-asyncio package to add support for async def test functions running under pytest. \n You can install these packages like so: \n pip install pytest pytest-asyncio \n If you are building an installable package you can add them as test dependencies to your setup.py module like this: \n setup(\n name=\"datasette-my-plugin\",\n # ...\n extras_require={\"test\": [\"pytest\", \"pytest-asyncio\"]},\n tests_require=[\"datasette-my-plugin[test]\"],\n) \n You can then install the test dependencies like so: \n pip install -e '.[test]' \n Then run the tests using pytest like so: \n pytest", "sections_fts": 186, "rank": null} {"rowid": 55, "title": "Setting up a Datasette test instance", "content": "The above example shows the easiest way to start writing tests against a Datasette instance: \n from datasette.app import Datasette\nimport pytest\n\n\n@pytest.mark.asyncio\nasync def test_plugin_is_installed():\n datasette = Datasette(memory=True)\n response = await datasette.client.get(\"/-/plugins.json\")\n assert response.status_code == 200 \n Creating a Datasette() instance like this as useful shortcut in tests, but there is one detail you need to be aware of. It's important to ensure that the async method .invoke_startup() is called on that instance. You can do that like this: \n datasette = Datasette(memory=True)\nawait datasette.invoke_startup() \n This method registers any startup(datasette) or prepare_jinja2_environment(env, datasette) plugins that might themselves need to make async calls. \n If you are using await datasette.client.get() and similar methods then you don't need to worry about this - Datasette automatically calls invoke_startup() the first time it handles a request.", "sections_fts": 186, "rank": null} {"rowid": 56, "title": "Using pdb for errors thrown inside Datasette", "content": "If an exception occurs within Datasette itself during a test, the response returned to your plugin will have a response.status_code value of 500. \n You can add pdb=True to the Datasette constructor to drop into a Python debugger session inside your test run instead of getting back a 500 response code. This is equivalent to running the datasette command-line tool with the --pdb option. \n Here's what that looks like in a test function: \n def test_that_opens_the_debugger_or_errors():\n ds = Datasette([db_path], pdb=True)\n response = await ds.client.get(\"/\") \n If you use this pattern you will need to run pytest with the -s option to avoid capturing stdin/stdout in order to interact with the debugger prompt.", "sections_fts": 186, "rank": null} {"rowid": 57, "title": "Using pytest fixtures", "content": "Pytest fixtures can be used to create initial testable objects which can then be used by multiple tests. \n A common pattern for Datasette plugins is to create a fixture which sets up a temporary test database and wraps it in a Datasette instance. \n Here's an example that uses the sqlite-utils library to populate a temporary test database. It also sets the title of that table using a simulated metadata.json configuration: \n from datasette.app import Datasette\nimport pytest\nimport sqlite_utils\n\n\n@pytest.fixture(scope=\"session\")\ndef datasette(tmp_path_factory):\n db_directory = tmp_path_factory.mktemp(\"dbs\")\n db_path = db_directory / \"test.db\"\n db = sqlite_utils.Database(db_path)\n db[\"dogs\"].insert_all(\n [\n {\"id\": 1, \"name\": \"Cleo\", \"age\": 5},\n {\"id\": 2, \"name\": \"Pancakes\", \"age\": 4},\n ],\n pk=\"id\",\n )\n datasette = Datasette(\n [db_path],\n metadata={\n \"databases\": {\n \"test\": {\n \"tables\": {\n \"dogs\": {\"title\": \"Some dogs\"}\n }\n }\n }\n },\n )\n return datasette\n\n\n@pytest.mark.asyncio\nasync def test_example_table_json(datasette):\n response = await datasette.client.get(\n \"/test/dogs.json?_shape=array\"\n )\n assert response.status_code == 200\n assert response.json() == [\n {\"id\": 1, \"name\": \"Cleo\", \"age\": 5},\n {\"id\": 2, \"name\": \"Pancakes\", \"age\": 4},\n ]\n\n\n@pytest.mark.asyncio\nasync def test_example_table_html(datasette):\n response = await datasette.client.get(\"/test/dogs\")\n assert \">Some dogs</h1>\" in response.text \n Here the datasette() function defines the fixture, which is than automatically passed to the two test functions based on pytest automatically matching their datasette function parameters. \n The @pytest.fixture(scope=\"session\") line here ensures the fixture is reused for the full pytest execution session. This means that the temporary database file will be created once and reused for each test. \n If you want to create that test database repeatedly for every individual test function, write the fixture function like this instead. You may want to do this if your plugin modifies the database contents in some way: \n @pytest.fixture\ndef datasette(tmp_path_factory):\n # This fixture will be executed repeatedly for every test\n ...", "sections_fts": 186, "rank": null} {"rowid": 58, "title": "Testing outbound HTTP calls with pytest-httpx", "content": "If your plugin makes outbound HTTP calls - for example datasette-auth-github or datasette-import-table - you may need to mock those HTTP requests in your tests. \n The pytest-httpx package is a useful library for mocking calls. It can be tricky to use with Datasette though since it mocks all HTTPX requests, and Datasette's own testing mechanism uses HTTPX internally. \n To avoid breaking your tests, you can return [\"localhost\"] from the non_mocked_hosts() fixture. \n As an example, here's a very simple plugin which executes an HTTP response and returns the resulting content: \n from datasette import hookimpl\nfrom datasette.utils.asgi import Response\nimport httpx\n\n\n@hookimpl\ndef register_routes():\n return [\n (r\"^/-/fetch-url$\", fetch_url),\n ]\n\n\nasync def fetch_url(datasette, request):\n if request.method == \"GET\":\n return Response.html(\n \"\"\"\n <form action=\"/-/fetch-url\" method=\"post\">\n <input type=\"hidden\" name=\"csrftoken\" value=\"{}\">\n <input name=\"url\"><input type=\"submit\">\n </form>\"\"\".format(\n request.scope[\"csrftoken\"]()\n )\n )\n vars = await request.post_vars()\n url = vars[\"url\"]\n return Response.text(httpx.get(url).text) \n Here's a test for that plugin that mocks the HTTPX outbound request: \n from datasette.app import Datasette\nimport pytest\n\n\n@pytest.fixture\ndef non_mocked_hosts():\n # This ensures httpx-mock will not affect Datasette's own\n # httpx calls made in the tests by datasette.client:\n return [\"localhost\"]\n\n\nasync def test_outbound_http_call(httpx_mock):\n httpx_mock.add_response(\n url=\"https://www.example.com/\",\n text=\"Hello world\",\n )\n datasette = Datasette([], memory=True)\n response = await datasette.client.post(\n \"/-/fetch-url\",\n data={\"url\": \"https://www.example.com/\"},\n )\n assert response.text == \"Hello world\"\n\n outbound_request = httpx_mock.get_request()\n assert (\n outbound_request.url == \"https://www.example.com/\"\n )", "sections_fts": 186, "rank": null} {"rowid": 59, "title": "Registering a plugin for the duration of a test", "content": "When writing tests for plugins you may find it useful to register a test plugin just for the duration of a single test. You can do this using pm.register() and pm.unregister() like this: \n from datasette import hookimpl\nfrom datasette.app import Datasette\nfrom datasette.plugins import pm\nimport pytest\n\n\n@pytest.mark.asyncio\nasync def test_using_test_plugin():\n class TestPlugin:\n __name__ = \"TestPlugin\"\n\n # Use hookimpl and method names to register hooks\n @hookimpl\n def register_routes(self):\n return [\n (r\"^/error$\", lambda: 1 / 0),\n ]\n\n pm.register(TestPlugin(), name=\"undo\")\n try:\n # The test implementation goes here\n datasette = Datasette()\n response = await datasette.client.get(\"/error\")\n assert response.status_code == 500\n finally:\n pm.unregister(name=\"undo\")", "sections_fts": 186, "rank": null} {"rowid": 60, "title": "Full-text search", "content": "SQLite includes a powerful mechanism for enabling full-text search against SQLite records. Datasette can detect if a table has had full-text search configured for it in the underlying database and display a search interface for filtering that table. \n Here's an example search : \n \n Datasette automatically detects which tables have been configured for full-text search.", "sections_fts": 186, "rank": null} {"rowid": 61, "title": "The table page and table view API", "content": "Table views that support full-text search can be queried using the ?_search=TERMS query string parameter. This will run the search against content from all of the columns that have been included in the index. \n Try this example: fara.datasettes.com/fara/FARA_All_ShortForms?_search=manafort \n SQLite full-text search supports wildcards. This means you can easily implement prefix auto-complete by including an asterisk at the end of the search term - for example: \n /dbname/tablename/?_search=rob* \n This will return all records containing at least one word that starts with the letters rob . \n You can also run searches against just the content of a specific named column by using _search_COLNAME=TERMS - for example, this would search for just rows where the name column in the FTS index mentions Sarah : \n /dbname/tablename/?_search_name=Sarah", "sections_fts": 186, "rank": null} {"rowid": 62, "title": "Advanced SQLite search queries", "content": "SQLite full-text search includes support for a variety of advanced queries , including AND , OR , NOT and NEAR . \n By default Datasette disables these features to ensure they do not cause errors or confusion for users who are not aware of them. You can disable this escaping and use the advanced queries by adding &_searchmode=raw to the table page query string. \n If you want to enable these operators by default for a specific table, you can do so by adding \"searchmode\": \"raw\" to the metadata configuration for that table, see Configuring full-text search for a table or view . \n If that option has been specified in the table metadata but you want to over-ride it and return to the default behavior you can append &_searchmode=escaped to the query string.", "sections_fts": 186, "rank": null} {"rowid": 63, "title": "Configuring full-text search for a table or view", "content": "If a table has a corresponding FTS table set up using the content= argument to CREATE VIRTUAL TABLE shown below, Datasette will detect it automatically and add a search interface to the table page for that table. \n You can also manually configure which table should be used for full-text search using query string parameters or Metadata . You can set the associated FTS table for a specific table and you can also set one for a view - if you do that, the page for that SQL view will offer a search option. \n Use ?_fts_table=x to over-ride the FTS table for a specific page. If the primary key was something other than rowid you can use ?_fts_pk=col to set that as well. This is particularly useful for views, for example: \n https://latest.datasette.io/fixtures/searchable_view?_fts_table=searchable_fts&_fts_pk=pk \n The fts_table metadata property can be used to specify an associated FTS table. If the primary key column in your table which was used to populate the FTS table is something other than rowid , you can specify the column to use with the fts_pk property. \n The \"searchmode\": \"raw\" property can be used to default the table to accepting SQLite advanced search operators, as described in Advanced SQLite search queries . \n Here is an example which enables full-text search (with SQLite advanced search operators) for a display_ads view which is defined against the ads table and hence needs to run FTS against the ads_fts table, using the id as the primary key: \n {\n \"databases\": {\n \"russian-ads\": {\n \"tables\": {\n \"display_ads\": {\n \"fts_table\": \"ads_fts\",\n \"fts_pk\": \"id\",\n \"searchmode\": \"raw\"\n }\n }\n }\n }\n}", "sections_fts": 186, "rank": null} {"rowid": 64, "title": "Searches using custom SQL", "content": "You can include full-text search results in custom SQL queries. The general pattern with SQLite search is to run the search as a sub-select that returns rowid values, then include those rowids in another part of the query. \n You can see the syntax for a basic search by running that search on a table page and then clicking \"View and edit SQL\" to see the underlying SQL. For example, consider this search for manafort is the US FARA database : \n /fara/FARA_All_ShortForms?_search=manafort \n If you click View and edit SQL you'll see that the underlying SQL looks like this: \n select\n rowid,\n Short_Form_Termination_Date,\n Short_Form_Date,\n Short_Form_Last_Name,\n Short_Form_First_Name,\n Registration_Number,\n Registration_Date,\n Registrant_Name,\n Address_1,\n Address_2,\n City,\n State,\n Zip\nfrom\n FARA_All_ShortForms\nwhere\n rowid in (\n select\n rowid\n from\n FARA_All_ShortForms_fts\n where\n FARA_All_ShortForms_fts match escape_fts(:search)\n )\norder by\n rowid\nlimit\n 101", "sections_fts": 186, "rank": null} {"rowid": 65, "title": "Enabling full-text search for a SQLite table", "content": "Datasette takes advantage of the external content mechanism in SQLite, which allows a full-text search virtual table to be associated with the contents of another SQLite table. \n To set up full-text search for a table, you need to do two things: \n \n \n Create a new FTS virtual table associated with your table \n \n \n Populate that FTS table with the data that you would like to be able to run searches against", "sections_fts": 186, "rank": null} {"rowid": 66, "title": "Configuring FTS using sqlite-utils", "content": "sqlite-utils is a CLI utility and Python library for manipulating SQLite databases. You can use it from Python code to configure FTS search, or you can achieve the same goal using the accompanying command-line tool . \n Here's how to use sqlite-utils to enable full-text search for an items table across the name and description columns: \n $ sqlite-utils enable-fts mydatabase.db items name description", "sections_fts": 186, "rank": null} {"rowid": 67, "title": "Configuring FTS using csvs-to-sqlite", "content": "If your data starts out in CSV files, you can use Datasette's companion tool csvs-to-sqlite to convert that file into a SQLite database and enable full-text search on specific columns. For a file called items.csv where you want full-text search to operate against the name and description columns you would run the following: \n $ csvs-to-sqlite items.csv items.db -f name -f description", "sections_fts": 186, "rank": null} {"rowid": 68, "title": "Configuring FTS by hand", "content": "We recommend using sqlite-utils , but if you want to hand-roll a SQLite full-text search table you can do so using the following SQL. \n To enable full-text search for a table called items that works against the name and description columns, you would run this SQL to create a new items_fts FTS virtual table: \n CREATE VIRTUAL TABLE \"items_fts\" USING FTS4 (\n name,\n description,\n content=\"items\"\n); \n This creates a set of tables to power full-text search against items . The new items_fts table will be detected by Datasette as the fts_table for the items table. \n Creating the table is not enough: you also need to populate it with a copy of the data that you wish to make searchable. You can do that using the following SQL: \n INSERT INTO \"items_fts\" (rowid, name, description)\n SELECT rowid, name, description FROM items; \n If your table has columns that are foreign key references to other tables you can include that data in your full-text search index using a join. Imagine the items table has a foreign key column called category_id which refers to a categories table - you could create a full-text search table like this: \n CREATE VIRTUAL TABLE \"items_fts\" USING FTS4 (\n name,\n description,\n category_name,\n content=\"items\"\n); \n And then populate it like this: \n INSERT INTO \"items_fts\" (rowid, name, description, category_name)\n SELECT items.rowid,\n items.name,\n items.description,\n categories.name\n FROM items JOIN categories ON items.category_id=categories.id; \n You can use this technique to populate the full-text search index from any combination of tables and joins that makes sense for your project.", "sections_fts": 186, "rank": null} {"rowid": 69, "title": "FTS versions", "content": "There are three different versions of the SQLite FTS module: FTS3, FTS4 and FTS5. You can tell which versions are supported by your instance of Datasette by checking the /-/versions page. \n FTS5 is the most advanced module but may not be available in the SQLite version that is bundled with your Python installation. Most importantly, FTS5 is the only version that has the ability to order by search relevance without needing extra code. \n If you can't be sure that FTS5 will be available, you should use FTS4.", "sections_fts": 186, "rank": null} {"rowid": 70, "title": "The Datasette Ecosystem", "content": "Datasette sits at the center of a growing ecosystem of open source tools aimed at making it as easy as possible to gather, analyze and publish interesting data. \n These tools are divided into two main groups: tools for building SQLite databases (for use with Datasette) and plugins that extend Datasette's functionality. \n The Datasette project website includes a directory of plugins and a directory of tools: \n \n \n Plugins directory on datasette.io \n \n \n Tools directory on datasette.io", "sections_fts": 186, "rank": null} {"rowid": 71, "title": "sqlite-utils", "content": "sqlite-utils is a key building block for the wider Datasette ecosystem. It provides a collection of utilities for manipulating SQLite databases, both as a Python library and a command-line utility. Features include: \n \n \n Insert data into a SQLite database from JSON, CSV or TSV, automatically creating tables with the correct schema or altering existing tables to add missing columns. \n \n \n Configure tables for use with SQLite full-text search, including creating triggers needed to keep the search index up-to-date. \n \n \n Modify tables in ways that are not supported by SQLite's default ALTER TABLE syntax - for example changing the types of columns or selecting a new primary key for a table. \n \n \n Adding foreign keys to existing database tables. \n \n \n Extracting columns of data into a separate lookup table.", "sections_fts": 186, "rank": null} {"rowid": 72, "title": "Dogsheep", "content": "Dogsheep is a collection of tools for personal analytics using SQLite and Datasette. The project provides tools like github-to-sqlite and twitter-to-sqlite that can import data from different sources in order to create a personal data warehouse. Personal Data Warehouses: Reclaiming Your Data is a talk that explains Dogsheep and demonstrates it in action.", "sections_fts": 186, "rank": null} {"rowid": 73, "title": "Deploying Datasette", "content": "The quickest way to deploy a Datasette instance on the internet is to use the datasette publish command, described in Publishing data . This can be used to quickly deploy Datasette to a number of hosting providers including Heroku, Google Cloud Run and Vercel. \n You can deploy Datasette to other hosting providers using the instructions on this page.", "sections_fts": 186, "rank": null} {"rowid": 74, "title": "Deployment fundamentals", "content": "Datasette can be deployed as a single datasette process that listens on a port. Datasette is not designed to be run as root, so that process should listen on a higher port such as port 8000. \n If you want to serve Datasette on port 80 (the HTTP default port) or port 443 (for HTTPS) you should run it behind a proxy server, such as nginx, Apache or HAProxy. The proxy server can listen on port 80/443 and forward traffic on to Datasette.", "sections_fts": 186, "rank": null} {"rowid": 75, "title": "Running Datasette using systemd", "content": "You can run Datasette on Ubuntu or Debian systems using systemd . \n First, ensure you have Python 3 and pip installed. On Ubuntu you can use sudo apt-get install python3 python3-pip . \n You can install Datasette into a virtual environment, or you can install it system-wide. To install system-wide, use sudo pip3 install datasette . \n Now create a folder for your Datasette databases, for example using mkdir /home/ubuntu/datasette-root . \n You can copy a test database into that folder like so: \n cd /home/ubuntu/datasette-root\ncurl -O https://latest.datasette.io/fixtures.db \n Create a file at /etc/systemd/system/datasette.service with the following contents: \n [Unit]\nDescription=Datasette\nAfter=network.target\n\n[Service]\nType=simple\nUser=ubuntu\nEnvironment=DATASETTE_SECRET=\nWorkingDirectory=/home/ubuntu/datasette-root\nExecStart=datasette serve . -h 127.0.0.1 -p 8000\nRestart=on-failure\n\n[Install]\nWantedBy=multi-user.target \n Add a random value for the DATASETTE_SECRET - this will be used to sign Datasette cookies such as the CSRF token cookie. You can generate a suitable value like so: \n $ python3 -c 'import secrets; print(secrets.token_hex(32))' \n This configuration will run Datasette against all database files contained in the /home/ubuntu/datasette-root directory. If that directory contains a metadata.yml (or .json ) file or a templates/ or plugins/ sub-directory those will automatically be loaded by Datasette - see Configuration directory mode for details. \n You can start the Datasette process running using the following: \n sudo systemctl daemon-reload\nsudo systemctl start datasette.service \n You will need to restart the Datasette service after making changes to its metadata.json configuration or adding a new database file to that directory. You can do that using: \n sudo systemctl restart datasette.service \n Once the service has started you can confirm that Datasette is running on port 8000 like so: \n curl 127.0.0.1:8000/-/versions.json\n# Should output JSON showing the installed version \n Datasette will not be accessible from outside the server because it is listening on 127.0.0.1 . You can expose it by instead listening on 0.0.0.0 , but a better way is to set up a proxy such as nginx - see Running Datasette behind a proxy .", "sections_fts": 186, "rank": null} {"rowid": 76, "title": "Running Datasette using OpenRC", "content": "OpenRC is the service manager on non-systemd Linux distributions like Alpine Linux and Gentoo . \n Create an init script at /etc/init.d/datasette with the following contents: \n #!/sbin/openrc-run\n\nname=\"datasette\"\ncommand=\"datasette\"\ncommand_args=\"serve -h 0.0.0.0 /path/to/db.db\"\ncommand_background=true\npidfile=\"/run/${RC_SVCNAME}.pid\" \n You then need to configure the service to run at boot and start it: \n rc-update add datasette\nrc-service datasette start", "sections_fts": 186, "rank": null} {"rowid": 77, "title": "Deploying using buildpacks", "content": "Some hosting providers such as Heroku , DigitalOcean App Platform and Scalingo support the Buildpacks standard for deploying Python web applications. \n Deploying Datasette on these platforms requires two files: requirements.txt and Procfile . \n The requirements.txt file lets the platform know which Python packages should be installed. It should contain datasette at a minimum, but can also list any Datasette plugins you wish to install - for example: \n datasette\ndatasette-vega \n The Procfile lets the hosting platform know how to run the command that serves web traffic. It should look like this: \n web: datasette . -h 0.0.0.0 -p $PORT --cors \n The $PORT environment variable is provided by the hosting platform. --cors enables CORS requests from JavaScript running on other websites to your domain - omit this if you don't want to allow CORS. You can add additional Datasette Settings options here too. \n These two files should be enough to deploy Datasette on any host that supports buildpacks. Datasette will serve any SQLite files that are included in the root directory of the application. \n If you want to build SQLite files or download them as part of the deployment process you can do so using a bin/post_compile file. For example, the following bin/post_compile will download an example database that will then be served by Datasette: \n wget https://fivethirtyeight.datasettes.com/fivethirtyeight.db \n simonw/buildpack-datasette-demo is an example GitHub repository showing a Datasette configuration that can be deployed to a buildpack-supporting host.", "sections_fts": 186, "rank": null} {"rowid": 78, "title": "Running Datasette behind a proxy", "content": "You may wish to run Datasette behind an Apache or nginx proxy, using a path within your existing site. \n You can use the base_url configuration setting to tell Datasette to serve traffic with a specific URL prefix. For example, you could run Datasette like this: \n datasette my-database.db --setting base_url /my-datasette/ -p 8009 \n This will run Datasette with the following URLs: \n \n \n http://127.0.0.1:8009/my-datasette/ - the Datasette homepage \n \n \n http://127.0.0.1:8009/my-datasette/my-database - the page for the my-database.db database \n \n \n http://127.0.0.1:8009/my-datasette/my-database/some_table - the page for the some_table table \n \n \n You can now set your nginx or Apache server to proxy the /my-datasette/ path to this Datasette instance.", "sections_fts": 186, "rank": null} {"rowid": 79, "title": "Nginx proxy configuration", "content": "Here is an example of an nginx configuration file that will proxy traffic to Datasette: \n daemon off;\n\nevents {\n worker_connections 1024;\n}\nhttp {\n server {\n listen 80;\n location /my-datasette {\n proxy_pass http://127.0.0.1:8009/my-datasette;\n proxy_set_header Host $host;\n }\n }\n} \n You can also use the --uds option to Datasette to listen on a Unix domain socket instead of a port, configuring the nginx upstream proxy like this: \n daemon off;\nevents {\n worker_connections 1024;\n}\nhttp {\n server {\n listen 80;\n location /my-datasette {\n proxy_pass http://datasette/my-datasette;\n proxy_set_header Host $host;\n }\n }\n upstream datasette {\n server unix:/tmp/datasette.sock;\n }\n} \n Then run Datasette with datasette --uds /tmp/datasette.sock path/to/database.db --setting base_url /my-datasette/ .", "sections_fts": 186, "rank": null} {"rowid": 80, "title": "Apache proxy configuration", "content": "For Apache , you can use the ProxyPass directive. First make sure the following lines are uncommented: \n LoadModule proxy_module lib/httpd/modules/mod_proxy.so\nLoadModule proxy_http_module lib/httpd/modules/mod_proxy_http.so \n Then add these directives to proxy traffic: \n ProxyPass /my-datasette/ http://127.0.0.1:8009/my-datasette/\nProxyPreserveHost On \n A live demo of Datasette running behind Apache using this proxy setup can be seen at datasette-apache-proxy-demo.datasette.io/prefix/ . The code for that demo can be found in the demos/apache-proxy directory. \n Using --uds you can use Unix domain sockets similar to the nginx example: \n ProxyPass /my-datasette/ unix:/tmp/datasette.sock|http://localhost/my-datasette/ \n The ProxyPreserveHost On directive ensures that the original Host: header from the incoming request is passed through to Datasette. Datasette needs this to correctly assemble links to other pages using the .absolute_url(request, path) method.", "sections_fts": 186, "rank": null} {"rowid": 81, "title": "Plugins", "content": "Datasette's plugin system allows additional features to be implemented as Python\n code (or front-end JavaScript) which can be wrapped up in a separate Python\n package. The underlying mechanism uses pluggy . \n See the Datasette plugins directory for a list of existing plugins, or take a look at the\n datasette-plugin topic on GitHub. \n Things you can do with plugins include: \n \n \n Add visualizations to Datasette, for example\n datasette-cluster-map and\n datasette-vega . \n \n \n Make new custom SQL functions available for use within Datasette, for example\n datasette-haversine and\n datasette-jellyfish . \n \n \n Define custom output formats with custom extensions, for example datasette-atom and\n datasette-ics . \n \n \n Add template functions that can be called within your Jinja custom templates,\n for example datasette-render-markdown . \n \n \n Customize how database values are rendered in the Datasette interface, for example\n datasette-render-binary and\n datasette-pretty-json . \n \n \n Customize how Datasette's authentication and permissions systems work, for example datasette-auth-tokens and\n datasette-permissions-sql .", "sections_fts": 186, "rank": null} {"rowid": 82, "title": "Installing plugins", "content": "If a plugin has been packaged for distribution using setuptools you can use the plugin by installing it alongside Datasette in the same virtual environment or Docker container. \n You can install plugins using the datasette install command: \n datasette install datasette-vega \n You can uninstall plugins with datasette uninstall : \n datasette uninstall datasette-vega \n You can upgrade plugins with datasette install --upgrade or datasette install -U : \n datasette install -U datasette-vega \n This command can also be used to upgrade Datasette itself to the latest released version: \n datasette install -U datasette \n These commands are thin wrappers around pip install and pip uninstall , which ensure they run pip in the same virtual environment as Datasette itself.", "sections_fts": 186, "rank": null} {"rowid": 83, "title": "One-off plugins using --plugins-dir", "content": "You can also define one-off per-project plugins by saving them as plugin_name.py functions in a plugins/ folder and then passing that folder to datasette using the --plugins-dir option: \n datasette mydb.db --plugins-dir=plugins/", "sections_fts": 186, "rank": null} {"rowid": 84, "title": "Deploying plugins using datasette publish", "content": "The datasette publish and datasette package commands both take an optional --install argument. You can use this one or more times to tell Datasette to pip install specific plugins as part of the process: \n datasette publish cloudrun mydb.db --install=datasette-vega \n You can use the name of a package on PyPI or any of the other valid arguments to pip install such as a URL to a .zip file: \n datasette publish cloudrun mydb.db \\\n --install=https://url-to-my-package.zip", "sections_fts": 186, "rank": null} {"rowid": 85, "title": "Seeing what plugins are installed", "content": "You can see a list of installed plugins by navigating to the /-/plugins page of your Datasette instance - for example: https://fivethirtyeight.datasettes.com/-/plugins \n You can also use the datasette plugins command: \n $ datasette plugins\n[\n {\n \"name\": \"datasette_json_html\",\n \"static\": false,\n \"templates\": false,\n \"version\": \"0.4.0\"\n }\n] \n [[[cog\nfrom datasette import cli\nfrom click.testing import CliRunner\nimport textwrap, json\ncog.out(\"\\n\")\nresult = CliRunner().invoke(cli.cli, [\"plugins\", \"--all\"])\n# cog.out() with text containing newlines was unindenting for some reason\ncog.outl(\"If you run ``datasette plugins --all`` it will include default plugins that ship as part of Datasette::\\n\")\nplugins = [p for p in json.loads(result.output) if p[\"name\"].startswith(\"datasette.\")]\nindented = textwrap.indent(json.dumps(plugins, indent=4), \" \")\nfor line in indented.split(\"\\n\"):\n cog.outl(line)\ncog.out(\"\\n\\n\") \n ]]] \n If you run datasette plugins --all it will include default plugins that ship as part of Datasette: \n [\n {\n \"name\": \"datasette.actor_auth_cookie\",\n \"static\": false,\n \"templates\": false,\n \"version\": null,\n \"hooks\": [\n \"actor_from_request\"\n ]\n },\n {\n \"name\": \"datasette.blob_renderer\",\n \"static\": false,\n \"templates\": false,\n \"version\": null,\n \"hooks\": [\n \"register_output_renderer\"\n ]\n },\n {\n \"name\": \"datasette.default_magic_parameters\",\n \"static\": false,\n \"templates\": false,\n \"version\": null,\n \"hooks\": [\n \"register_magic_parameters\"\n ]\n },\n {\n \"name\": \"datasette.default_menu_links\",\n \"static\": false,\n \"templates\": false,\n \"version\": null,\n \"hooks\": [\n \"menu_links\"\n ]\n },\n {\n \"name\": \"datasette.default_permissions\",\n \"static\": false,\n \"templates\": false,\n \"version\": null,\n \"hooks\": [\n \"permission_allowed\"\n ]\n },\n {\n \"name\": \"datasette.facets\",\n \"static\": false,\n \"templates\": false,\n \"version\": null,\n \"hooks\": [\n \"register_facet_classes\"\n ]\n },\n {\n \"name\": \"datasette.filters\",\n \"static\": false,\n \"templates\": false,\n \"version\": null,\n \"hooks\": [\n \"filters_from_request\"\n ]\n },\n {\n \"name\": \"datasette.forbidden\",\n \"static\": false,\n \"templates\": false,\n \"version\": null,\n \"hooks\": [\n \"forbidden\"\n ]\n },\n {\n \"name\": \"datasette.handle_exception\",\n \"static\": false,\n \"templates\": false,\n \"version\": null,\n \"hooks\": [\n \"handle_exception\"\n ]\n },\n {\n \"name\": \"datasette.publish.cloudrun\",\n \"static\": false,\n \"templates\": false,\n \"version\": null,\n \"hooks\": [\n \"publish_subcommand\"\n ]\n },\n {\n \"name\": \"datasette.publish.heroku\",\n \"static\": false,\n \"templates\": false,\n \"version\": null,\n \"hooks\": [\n \"publish_subcommand\"\n ]\n },\n {\n \"name\": \"datasette.sql_functions\",\n \"static\": false,\n \"templates\": false,\n \"version\": null,\n \"hooks\": [\n \"prepare_connection\"\n ]\n }\n] \n [[[end]]] \n You can add the --plugins-dir= option to include any plugins found in that directory.", "sections_fts": 186, "rank": null} {"rowid": 86, "title": "Plugin configuration", "content": "Plugins can have their own configuration, embedded in a Metadata file. Configuration options for plugins live within a \"plugins\" key in that file, which can be included at the root, database or table level. \n Here is an example of some plugin configuration for a specific table: \n {\n \"databases\": {\n \"sf-trees\": {\n \"tables\": {\n \"Street_Tree_List\": {\n \"plugins\": {\n \"datasette-cluster-map\": {\n \"latitude_column\": \"lat\",\n \"longitude_column\": \"lng\"\n }\n }\n }\n }\n }\n }\n} \n This tells the datasette-cluster-map column which latitude and longitude columns should be used for a table called Street_Tree_List inside a database file called sf-trees.db .", "sections_fts": 186, "rank": null} {"rowid": 87, "title": "Secret configuration values", "content": "Any values embedded in metadata.json will be visible to anyone who views the /-/metadata page of your Datasette instance. Some plugins may need configuration that should stay secret - API keys for example. There are two ways in which you can store secret configuration values. \n As environment variables . If your secret lives in an environment variable that is available to the Datasette process, you can indicate that the configuration value should be read from that environment variable like so: \n {\n \"plugins\": {\n \"datasette-auth-github\": {\n \"client_secret\": {\n \"$env\": \"GITHUB_CLIENT_SECRET\"\n }\n }\n }\n} \n As values in separate files . Your secrets can also live in files on disk. To specify a secret should be read from a file, provide the full file path like this: \n {\n \"plugins\": {\n \"datasette-auth-github\": {\n \"client_secret\": {\n \"$file\": \"/secrets/client-secret\"\n }\n }\n }\n} \n If you are publishing your data using the datasette publish family of commands, you can use the --plugin-secret option to set these secrets at publish time. For example, using Heroku you might run the following command: \n $ datasette publish heroku my_database.db \\\n --name my-heroku-app-demo \\\n --install=datasette-auth-github \\\n --plugin-secret datasette-auth-github client_id your_client_id \\\n --plugin-secret datasette-auth-github client_secret your_client_secret \n This will set the necessary environment variables and add the following to the deployed metadata.json : \n {\n \"plugins\": {\n \"datasette-auth-github\": {\n \"client_id\": {\n \"$env\": \"DATASETTE_AUTH_GITHUB_CLIENT_ID\"\n },\n \"client_secret\": {\n \"$env\": \"DATASETTE_AUTH_GITHUB_CLIENT_SECRET\"\n }\n }\n }\n}", "sections_fts": 186, "rank": null} {"rowid": 88, "title": "Settings", "content": "", "sections_fts": 186, "rank": null} {"rowid": 89, "title": "Using --setting", "content": "Datasette supports a number of settings. These can be set using the --setting name value option to datasette serve . \n You can set multiple settings at once like this: \n datasette mydatabase.db \\\n --setting default_page_size 50 \\\n --setting sql_time_limit_ms 3500 \\\n --setting max_returned_rows 2000", "sections_fts": 186, "rank": null} {"rowid": 90, "title": "Configuration directory mode", "content": "Normally you configure Datasette using command-line options. For a Datasette instance with custom templates, custom plugins, a static directory and several databases this can get quite verbose: \n $ datasette one.db two.db \\\n --metadata=metadata.json \\\n --template-dir=templates/ \\\n --plugins-dir=plugins \\\n --static css:css \n As an alternative to this, you can run Datasette in configuration directory mode. Create a directory with the following structure: \n # In a directory called my-app:\nmy-app/one.db\nmy-app/two.db\nmy-app/metadata.json\nmy-app/templates/index.html\nmy-app/plugins/my_plugin.py\nmy-app/static/my.css \n Now start Datasette by providing the path to that directory: \n $ datasette my-app/ \n Datasette will detect the files in that directory and automatically configure itself using them. It will serve all *.db files that it finds, will load metadata.json if it exists, and will load the templates , plugins and static folders if they are present. \n The files that can be included in this directory are as follows. All are optional. \n \n \n *.db (or *.sqlite3 or *.sqlite ) - SQLite database files that will be served by Datasette \n \n \n metadata.json - Metadata for those databases - metadata.yaml or metadata.yml can be used as well \n \n \n inspect-data.json - the result of running datasette inspect *.db --inspect-file=inspect-data.json from the configuration directory - any database files listed here will be treated as immutable, so they should not be changed while Datasette is running \n \n \n settings.json - settings that would normally be passed using --setting - here they should be stored as a JSON object of key/value pairs \n \n \n templates/ - a directory containing Custom templates \n \n \n plugins/ - a directory containing plugins, see Writing one-off plugins \n \n \n static/ - a directory containing static files - these will be served from /static/filename.txt , see Serving static files", "sections_fts": 186, "rank": null} {"rowid": 91, "title": "Settings", "content": "The following options can be set using --setting name value , or by storing them in the settings.json file for use with Configuration directory mode .", "sections_fts": 186, "rank": null} {"rowid": 92, "title": "default_allow_sql", "content": "Should users be able to execute arbitrary SQL queries by default? \n Setting this to off causes permission checks for execute-sql to fail by default. \n datasette mydatabase.db --setting default_allow_sql off \n There are two ways to achieve this: the other is to add \"allow_sql\": false to your metadata.json file, as described in Controlling the ability to execute arbitrary SQL . This setting offers a more convenient way to do this.", "sections_fts": 186, "rank": null} {"rowid": 93, "title": "default_page_size", "content": "The default number of rows returned by the table page. You can over-ride this on a per-page basis using the ?_size=80 query string parameter, provided you do not specify a value higher than the max_returned_rows setting. You can set this default using --setting like so: \n datasette mydatabase.db --setting default_page_size 50", "sections_fts": 186, "rank": null} {"rowid": 94, "title": "sql_time_limit_ms", "content": "By default, queries have a time limit of one second. If a query takes longer than this to run Datasette will terminate the query and return an error. \n If this time limit is too short for you, you can customize it using the sql_time_limit_ms limit - for example, to increase it to 3.5 seconds: \n datasette mydatabase.db --setting sql_time_limit_ms 3500 \n You can optionally set a lower time limit for an individual query using the ?_timelimit=100 query string argument: \n /my-database/my-table?qSpecies=44&_timelimit=100 \n This would set the time limit to 100ms for that specific query. This feature is useful if you are working with databases of unknown size and complexity - a query that might make perfect sense for a smaller table could take too long to execute on a table with millions of rows. By setting custom time limits you can execute queries \"optimistically\" - e.g. give me an exact count of rows matching this query but only if it takes less than 100ms to calculate.", "sections_fts": 186, "rank": null} {"rowid": 95, "title": "max_returned_rows", "content": "Datasette returns a maximum of 1,000 rows of data at a time. If you execute a query that returns more than 1,000 rows, Datasette will return the first 1,000 and include a warning that the result set has been truncated. You can use OFFSET/LIMIT or other methods in your SQL to implement pagination if you need to return more than 1,000 rows. \n You can increase or decrease this limit like so: \n datasette mydatabase.db --setting max_returned_rows 2000", "sections_fts": 186, "rank": null} {"rowid": 96, "title": "num_sql_threads", "content": "Maximum number of threads in the thread pool Datasette uses to execute SQLite queries. Defaults to 3. \n datasette mydatabase.db --setting num_sql_threads 10 \n Setting this to 0 turns off threaded SQL queries entirely - useful for environments that do not support threading such as Pyodide .", "sections_fts": 186, "rank": null} {"rowid": 97, "title": "allow_facet", "content": "Allow users to specify columns they would like to facet on using the ?_facet=COLNAME URL parameter to the table view. \n This is enabled by default. If disabled, facets will still be displayed if they have been specifically enabled in metadata.json configuration for the table. \n Here's how to disable this feature: \n datasette mydatabase.db --setting allow_facet off", "sections_fts": 186, "rank": null} {"rowid": 98, "title": "default_facet_size", "content": "The default number of unique rows returned by Facets is 30. You can customize it like this: \n datasette mydatabase.db --setting default_facet_size 50", "sections_fts": 186, "rank": null} {"rowid": 99, "title": "facet_time_limit_ms", "content": "This is the time limit Datasette allows for calculating a facet, which defaults to 200ms: \n datasette mydatabase.db --setting facet_time_limit_ms 1000", "sections_fts": 186, "rank": null} {"rowid": 100, "title": "facet_suggest_time_limit_ms", "content": "When Datasette calculates suggested facets it needs to run a SQL query for every column in your table. The default for this time limit is 50ms to account for the fact that it needs to run once for every column. If the time limit is exceeded the column will not be suggested as a facet. \n You can increase this time limit like so: \n datasette mydatabase.db --setting facet_suggest_time_limit_ms 500", "sections_fts": 186, "rank": null}