Fix minor error in documentation.

LICENSE

@@ -1,8 +1,6 @@
-No Nazis, otherwise:
-
 MIT License
 
-Copyright (c) 2023-2024 Aldercone Studio Collective
+Copyright (c) 2018 Cas Rusnov
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

MANIFEST.in

@@ -0,0 +1 @@
+include pixywerk2/defaults/*.yaml

README.md

@@ -1,5 +1,5 @@
-# Heckweasel #
+# Pixywerk #
 
-Heckweasel is a site compiler engineered like a metadata-based CMS with a template rendering system. Underneath it uses
+PixyWerk2 is a site compiler engineered like a metadata-based CMS with a template rendering system. Underneath it uses
 Jinja2 templates to provide programmability, and a structured metadata system, along with processors to convert
 user-friendly files such as Markdown and RST into HTML with templates.

READMES/METADATA.md

@@ -15,8 +15,6 @@ On-disk meatdata is stored as a file along side the non-metadata file with the e
 
 All files define the following keys by default:
 
-relpath
-:  The relative path to the root of the site, useful for prepending to image `src=` and other resource paths such as CSS files and fonts in order to maintain locally viewable output.
 file_name
 :  The local path of the file
 file_path
@@ -62,14 +60,6 @@ author_email
 site_root
 :  The full URL for the root of this web site used for links and whatnot, with ending slash.
 
-Special Keys that can be defined, these change the processing in predictable ways:
-
-type
-: Define that the file that this metadata is applied to as a specific type from the type mapping table. Useful values are `passthrough` and `templatable` with obvious outcomes.
-wildcard_metadata
-: Define a dictionary of file globs (patterns which match files such as `*.txt`), with the value being a dictionary of additional metadata to apply to the matched files. This is generally
-defined at the top level of the project to make certain file patterns treated as special without having to give them their own metadata.
-
 
 ## CACHING STRATEGY ##
 

READMES/project-layout.md

@@ -1,13 +1,11 @@
 # Project Layout #
 
-It is recommended that in general your project for Heckweasel site be layed out like:
+It is recommended that in general your project for PixyWerk2 site be layed out like:
 ```
 project_top/
    Makefile              - Convenient for building your site
    src/                  - All "source" pages are contained in here.
      .meta               - Top-level default metadata is set here
-	 index.cont          - The content part of the index page
-	 index.cont.meta     - A metadata json file for the index, specifically.
      templates/          - Templates go in here
        default.jinja2    - Default template that will be used if none are specified
    publish/              - The path the build process will create, where the post-processed files go.
@@ -21,7 +19,7 @@ site. Something as simple as:
 
 ```
 build: src/templates/* src/*
-	python -mheckweasel src publish
+	python -mpixywerk2 src publish
 ```
 
 ## src/ ##
@@ -68,4 +66,4 @@ A simple default.jinja2 example:
 
 ## publish/ ##
 
-This is arbitrary, and will be created by heckweasel at build time, but it will be the root path that should be published to your web server.
+This is arbitrary, and will be created by pixywerk at build time, but it will be the root path that should be published to your web server.

TODO.md

@@ -1,34 +1,8 @@
 # TODO #
 
+* Pygments pretty printing of source code et al. including exposing that to the template API (`pygment_format(get_file_content('whatever.py'))`).
 * Smart CSS things (fill in the processors)
-* Project global defines, parameters.
-* pre- and post-scripts that will be run from __main__, either some shipped with heckweasel or project-level.
+
+# Maybe #
+
 * Library of template modules? ATOM et al.
-* Some off the shelf website templates and a template manager.
-* Live refreshing server thing which maps a heckweasel tree into a web server's memory and updates on change.
-* https://github.com/Python-Markdown/markdown/wiki/Third-Party-Extensions
- * add markdown_link_attr_modifier extension
- * add figureAltCaption extension
- * add qrcode extension
-* Add support to define macros or whatever for Jinja, or to include generic stanzas in any output so adding macros won't mean repeatedly including them.
-
-* It'd be good to generate a dependency tree and only recompile things based on changes, like makefile-like behavior.
-
-* Fragments which would be blobs of mechanics like rss feed, thumbnail links, etc. They would be virtual files and other changes to processing
-  chains and project contents. `python -mheckweasel --fragment=rss,config=foo.meta` etc.
-  
-* Run commands as part of processing chains
-
-* Project level processing chain overrides in the .meta or whatever.
-* Project settings in separate file from .meta that would basically do .meta stuff. Like global meta + config in a top.heck file by default and overridable by a parameter. Maybe
-  a nice default filename that doesn't start with . (whereas .meta or .heck is the current base metadata)
-
-* Handle the fact that HECKformat metadata is always a list in a more elegent way. We have some hacks in place where scalar values are expected, but if a project mixes
-  metadata formats it gets a bit messy.
-
-* Provide a database-like interface to the global metadata tree, so that metameta page templates could query the database to assemble indexes and whatnot without looping over the actual files.
-
-* add a version requirement system to projects so if we start breaking things we can use versioning
-* add a flag to __main__ to ignore errors even at the end 
-* add a flag to __main__ to exit on first error rather than waiting til the end.
-

demo/.meta

@@ -0,0 +1,7 @@
+{
+"site_root":"https://example.com",
+"title":"Test Metadata",
+"author": "Test User",
+"author_email": "test_user@example.com",
+"uuid_oid_root": "pixywerk-demo"
+}

demo/atom.xml

@@ -0,0 +1,33 @@
+<?xml version="1.0" encoding="utf-8"?>
+
+<feed xmlns="http://www.w3.org/2005/Atom">
+
+	<title>{{ metadata.title }}</title>
+	<subtitle>{{ metadata.subtitle }}</subtitle>
+	<link href="{{ metadata.site_root }}/{{ metadata.file_name }}" rel="self" />
+	<link href="{{ metadata.site_root }}" />
+	<id>urn:uuid:{{ metadata.uuid }}</id>
+	<updated>{{ get_time_iso8601(metadata['build-time']) }}</updated>
+
+	{% set posts = get_file_list('blog_posts/*.cont') %}
+	{% for post in posts %}
+	{% set post_meta = get_file_metadata(post['file_path']) %}
+	<entry>
+		<title>{{ post_meta.title }}</title>
+		<link href="{{ metadata.site_root }}/{{post_meta.file_path}}" />
+		<id>urn:uuid:{{ post_meta.uuid }}</id>
+		<updated>{{ get_time_iso8601(post_meta.stat.mtime) }}</updated>
+		<summary>{{post_meta.summary }}</summary>
+		<!-- this would be the snippet, more than summary chunk -->
+		<!-- <content type="xhtml"> -->
+		<!-- 	<div xmlns="http://www.w3.org/1999/xhtml"> -->
+		<!-- 		<p>{{ post_meta.summary }}</p> -->
+		<!-- 	</div> -->
+		<!-- </content> -->
+		<author>
+			<name>{{ post_meta.author }}</name>
+			<email>{{ post_meta.author_email }}</email>
+		</author>
+	</entry>
+	{% endfor %}
+</feed>

demo/atom.xml.meta

@@ -0,0 +1,5 @@
+{
+"type": "templatable",
+"title": "Test RSS Feed",
+"subtitle": "Some Subtitle"
+}

demo/blog_posts/anotherpost.cont

@@ -0,0 +1,5 @@
+Some more post
+
+
+la la la
+

demo/blog_posts/anotherpost.cont.meta

@@ -0,0 +1,4 @@
+{
+"title":"Another Post(tm)",
+"summary":"Yet another post"
+}

demo/blog_posts/test.cont

@@ -0,0 +1 @@
+Some content.

demo/blog_posts/test.cont.meta

@@ -0,0 +1,4 @@
+{
+"title":"Test.cont",
+"summary":"Some empty test content"
+}

demo/foo.cont

@@ -0,0 +1 @@
+yo fresh

demo/foo.cont.meta

@@ -0,0 +1,5 @@
+{
+"foo":"bar",
+"title":"A title",
+"summary":"Just a post."
+}

demo/index.cont

@@ -0,0 +1,19 @@
+<h1>Index of all content</h1>
+{% for f in get_file_list('*', sort_order='file_name') %}
+<a href="{{ get_file_name(f['file_name']) }}">{{get_file_name(f['file_name'])}}</a>
+{% endfor %}
+
+<p>Including foo.cont.meta:
+<pre>
+{{ get_file_content('foo.cont.meta') }}
+</pre>
+</p>
+
+<h1>Metadata</h1>
+<table class="metadata">
+<tr><th>key</th><th>value</th></tr>
+{% set metadata = get_file_metadata('foo.cont') %}
+{% for k in metadata.keys() %}
+<tr><td>{{k}}</td><td>{{metadata[k]}}</td></tr>
+{% endfor %}
+</table>

demo/passthrough.md

@@ -0,0 +1,9 @@
+# README #
+
+This is a test of the emergency compiled HTML system. This is only a *test*.
+
+[Foo!](foo.html)
+
+{% for i in range(100) %}
+* {{ i }}
+{% endfor %}

demo/passthrough.md.meta

@@ -0,0 +1,3 @@
+{
+"pragma":["no-proc"]
+}

demo/readme.md

@@ -0,0 +1,9 @@
+# README #
+
+This is a test of the emergency compiled HTML system. This is only a *test*.
+
+[Foo!](foo.html)
+
+{% for i in range(100) %}
+* {{ i }}
+{% endfor %}

demo/readme.md.meta

@@ -0,0 +1,3 @@
+{
+"title":"Yo, markdown"
+}

demo/templates/debug.jinja2

@@ -0,0 +1,32 @@
+<!DOCTYPE html>
+<head>
+<title>Debug for {{path}}</title>
+<style type="text/css">
+table { border: 1px solid black; }
+div { border: 1px solid black; }
+td { border: 1px solid black; }
+</style>
+</head>
+<body>
+<p>{{path}}</p>
+<h1>Content</h1>
+<div class="content">
+{{content}}
+</div>
+
+<h1>Environment</h1>
+<table class="environment">
+<tr><th>key</th><th>value</th></tr>
+{% for k in environ.keys() %}
+<tr><td>{{k}}</td><td>{{environ[k]}}</td></tr>
+{% endfor %}
+</table>
+
+<h1>Metadata</h1>
+<table class="metadata">
+<tr><th>key</th><th>value</th></tr>
+{% for k in metadata.keys() %}
+<tr><td>{{k}}</td><td>{{metadata[k]}}</td></tr>
+{% endfor %}
+</table>
+</body>

demo/templates/default-fs.jinja2

@@ -0,0 +1,6 @@
+<table class="werk-file-list">
+<tr class="werk-file-list-head"><th>file</th><th>type</th><th>size</th><th>last change</th></tr>
+{% for f in files.keys() %}
+<tr class="werk-file-list-item"><td><a href="/{{files[f].relpath}}">{{f}}</a></td><td>{{files[f].type}}</td><td>{{files[f].size}}</td><td>{{files[f].ctime | date}}</td></tr>
+{% endfor %}
+</table>

demo/templates/default.jinja2

@@ -0,0 +1,13 @@
+<!DOCTYPE html>
+<head>
+<title>{{metadata.title}}</title>
+<style type="text/css">
+table { border: 1px solid black; }
+div { border: 1px solid black; }
+td { border: 1px solid black; }
+</style>
+</head>
+<body>
+{{content}}
+</body>
+</html>

docs/index.html

@@ -1,65 +0,0 @@
-<h1>HECKWEASEL documentation!</h1>
-
-<p>Welcome to the index for HECKWEASEL Documentation. In this directory you&rsquo;ll find a bunch of files but this is the introduction you need to understanding the way heckweasel works and how to use it.</p>
-
-<h2>Introduction Part 2: What the hyeck is Heckweasel?!</h2>
-
-<p>Heckweasel is a website compiler framework. Primarily it allows the creation of web site using a collection of flat files which are in a maintainable form, producing the less maintainable formats that web browsers use.</p>
-
-<p>The flat files in a heckweasel project are just a directory of files like any other. There is a default directory structure for projects but that isn&rsquo;t important right now.</p>
-
-<p>Heckweasel projects generally take the form of a collection of one or more templates and a collection of one or more files that are filled into the templates. Pervasively, heckweasel draws a distinction between the contents of a web page and the template it gets put into. You can think of the template, as generally used by heckweasel, as a sort of picture frame into which your content is placed. The content itself may be implemented as one of several popular formats such as Markdown and HTML. Also of note is that there are sort of two routes from heckweeasel input to heckweasel output, one route is through the template system and the other route merely copies the input to the output.</p>
-
-<p>Another important detail about heckweasel is metadata. Every item in the heckweasel project (thus, every file in the heckweasel project directory) has a collection of <em>metadata</em> associated with it, such as its file name, creation time, and other objective information, but also any arbitrary information about it such as its title, a short description, thumbnails or whatever. It&rsquo;s also important to note that the <strong>content</strong> of a file counts as metadata, and is stored the same way inside of heckweasel&rsquo;s way of looking at the files. Metadata is stored with the file as <em>filename</em>.meta and directories contain metadata in the file called .meta. Metadata is also inherited! So setting a template in a directory&rsquo;s metadata will apply to all of the contents of that directory. Metadata is all in a JSON format called JStyleSon, which is JSON except you can have comments in it. All of these metadata are accessable from the templates, which leads to&hellip;</p>
-
-<p>The final important detail about heckweasel is that it, at is core, uses a programmable template system called Jinja. Jinja allows a lot, and I mean a <em>lot</em> of flexability in the way that the output is produced, giving complete programmability. This allows templates (and pages, for that matter) to contain programmable outcomes such as showing a list of all blog entries (each of which would be a separate file), or making a thumbnail gallery from a collection of pictures, or generating an RSS feed from all of the contents of the site. This also allows the website design to be broken into parts such that commonly-used patterns can be merely included in the file rather than being written repeatedly (although normally this function done with the page templates).</p>
-
-<h2>Just the very Basic Heckweasel Project</h2>
-
-<p>So with all of that said, the most basic possible heckweasel project that is actually functional would be something like a page template, and a content file called index. Heckweasel operates on an input directory and outputs to an output directory. This is admittedly not a normal use case since it doesn&rsquo;t benifit much from the elaborate system underneath, but it gets the idea across.</p>
-
-<p>So you have your project directory <code>mywebsite</code>; inside we can have the directories <code>source</code> and <code>publish</code>, and various files, and well here&rsquo;s a picture:</p>
-
-<ul>
-<li><strong>mywebsite</strong>
-
-<ul>
-<li><strong>source</strong>
-
-<ul>
-<li><em>.meta</em></li>
-<li><strong>templates</strong>
-
-<ul>
-<li><em>default.jinja</em></li>
-</ul>
-</li>
-<li><em>index.md</em></li>
-<li><em>index.md.meta</em></li>
-</ul>
-</li>
-<li><strong>publish</strong></li>
-</ul>
-</li>
-</ul>
-
-
-<p>To explain the various files:</p>
-
-<h3><em>.meta</em></h3>
-
-<p>This file is a JSON file containing project-wide metadata. Usually this would be metadata that applies, by default, to all files. Some things that affect the way Heckweasel processes files would be <code>template</code> which would set the default template to put content into and <code>templates</code> which would set the directory to look for templates in. By custom we also may want to set the title, author and other things like that which we may want to fill into the output files. We also put things like the eventual published address for the site (<code>site_root</code>).</p>
-
-<p>Example .meta file:</p>
-
-<p>```json</p>
-
-<p>{
-    &ldquo;site_root&rdquo;: &ldquo;https://website.me&rdquo;,
-    &ldquo;author&rdquo;: &ldquo;Very Nice Person&rdquo;,
-    &ldquo;title&rdquo;: &ldquo;My Website&rdquo;
-}</p>
-
-<p>```</p>
-
-<h3><em>default.jinja</em></h3>

docs/index.md

@@ -1,151 +0,0 @@
-# HECKWEASEL documentation!
-
-Welcome to the index for HECKWEASEL Documentation. In this directory you'll find a bunch of files but this is the introduction you need to understanding the way heckweasel works and how to use it. You wouldn't web a site.
-
-## Introduction: TL;DR
-
-Heckweasel compiles a set of files into a website.
-
-There is your website **template**, separate files that are the **content** of your website (such as a blog post, an image, etc), and json files that are the **metadata** for each content file. These get compiled together into static web pages.
-
-There's a lot more to it, and it is entirely programmable, but basically that's it.
-
-
-## Introduction: What the hyeck is Heckweasel?!
-
-Heckweasel is a website compiler framework. Primarily it allows the creation of web site using a collection of flat files which are in a maintainable form, producing the less maintainable formats that web browsers use.
-
-The flat files in a heckweasel project are just a directory of files like any other. There is a default directory structure for projects but that isn't important right now.
-
-Heckweasel projects generally take the form of a collection of one or more templates and a collection of one or more files that are filled into the templates. Pervasively, heckweasel draws a distinction between the contents of a web page and the template it gets put into. You can think of the template, as generally used by heckweasel, as a sort of picture frame into which your content is placed. The content itself may be implemented as one of several popular formats such as Markdown and HTML. Also of note is that there are sort of two routes from heckweeasel input to heckweasel output, one route is through the template system and the other route merely copies the input to the output.
-
-Another important detail about heckweasel is metadata. Every item in the heckweasel project (thus, every file in the heckweasel project directory) has a collection of *metadata* associated with it, such as its file name, creation time, and other objective information, but also any arbitrary information about it such as its title, a short description, thumbnails or whatever. It's also important to note that the **content** of a file counts as metadata, and is stored the same way inside of heckweasel's way of looking at the files. Metadata is stored with the file as *filename*.meta and directories contain metadata in the file called .meta. Metadata is also inherited! So setting a template in a directory's metadata will apply to all of the contents of that directory. Metadata is all in a JSON format called JStyleSon, which is JSON except you can have comments in it. All of these metadata are accessable from the templates, which leads to...
-
-The final important detail about heckweasel is that it, at is core, uses a programmable template system called Jinja. Jinja allows a lot, and I mean a *lot* of flexability in the way that the output is produced, giving complete programmability. This allows templates (and pages, for that matter) to contain programmable outcomes such as showing a list of all blog entries (each of which would be a separate file), or making a thumbnail gallery from a collection of pictures, or generating an RSS feed from all of the contents of the site. This also allows the website design to be broken into parts such that commonly-used patterns can be merely included in the file rather than being written repeatedly (although normally this function done with the page templates).
-
-
-## Glossary
-
-- **template**
-    - A -link-Jinja2 file which gets filled in with your content
-- **content**
-    - The content which gets filled into templates to produce pages
-- **metadata**
-    - Extra variables or values associated with content, which can be used to modify the way template works and do other tricks
-
-
-
-## Just the very Basic Heckweasel Project
-
-So with all of that said, the most basic possible heckweasel project that is actually functional would be something like a page template, and a content file called index. Heckweasel operates on an input directory and outputs to an output directory. This is admittedly not a normal use case since it doesn't benifit much from the elaborate system underneath, but it gets the idea across.
-
-So you have your project directory `mywebsite`; inside we can have the directories `source` and `publish`, and various files, and well here's a picture:
-
-- __mywebsite__
-    - __source__
-        - *.meta*
-        - __templates__
-            - *default.jinja*
-        - *index.md*
-        - *index.md.meta*
-    - __publish__
-  
-  
-To explain the various files:
-
-
-### *.meta*
-
-This file is a JSON file containing project-wide metadata. Usually this would be metadata that applies, by default, to all files. Some things that affect the way Heckweasel processes files would be `template` which would set the default template to put content into and `templates` which would set the directory to look for templates in. By custom we also may want to set the title, author and other things like that which we may want to fill into the output files. We also put things like the eventual published address for the site (`site_root`).
-
-Example .meta file:
-
-
-```json
-{
-    "site_root": "https://website.me",
-    "author": "Very Nice Person",
-    "title": "My Website"
-}
-
-```
-
-### *default.jinja*
-
-This is the default template. Heckweasel will look for `templates/default.jinja` unless another templates directory and template are specified. Jinja templates might output any kind of text file you want, but usually we put HTML inside them. Here's an example `default.jinja` that makes a barely functional web page but we'll explain more later:
-
-```jinja2
-<!DOCTYPE html>
-<html>
-<head>
-<title>{{ metadata.title }}</title>
-</head>
-<body>
-{{content}}
-</body>
-</html>
-```
-
-The main thing to notice is that this is a very simple HTML file. It does the bare minimum to render in a browser. The next thing to notice are all of the `{}` things. Those are Jinja commands. A `{{}}` containing a name will fill that name from the variables set in the Jinja environment. In Heckweasel the main two things are `content` and `metadata`. `Metadata` contains the metadata set via the `.meta` and other sources as discussed above. The new thing here is `content`, which is the *contents* of the page! As discussed above, the contents and template are considered separately, and so the page contents are filled into the template where the `{{content}}` tag is! You can also see that the title of the page is set based on the page's `title` metadata. We'll discuss this more in the next section.
-
-Another interesting thing is, any styling that should be applied to the whole website, to a particular page type, or whatever goes in templates. For example this is where you'd include the site-wide CSS sheet for this site, and it would apply that style to all the pages (we'll discuss this more in a future section).
-
-
-### *index.md*
-
-This is the contents of the page that will eventually become `index.html` when heckweasel is done with it. Notice it is `.md` which means markdown, a user-friendly markup format - heckweasel will convert this to an HTML fragment and fill in the template's `content` with the result, producing `index.html`. This is how the magic happens! The contents of this file could be something as simple as:
-
-```markdown
-# Welcome!
-
-Hello this is my website! Hi!
-```
-
-### *index.md.meta*
-
-This contains the metadata specific to `index.md`. It can be left out if there isn't any specific metadata, but it's useful to make even an empty one for future reference. An example use of this is to set different title for each page.
-
-Example:
-
-```json
-{
-  "title": "Welcome to my Home Page"
-}
-```
-
-### Rolling it all together
-
-Given the above tree, from the command line in the `mywebsite` directory, to compile this would be as simple as :
-
-```bash
-$ python -mheckweasel source publish
-```
-
-This would produce, in the `publish` directory, `index.html`, which would have contents like:
-
-```html
-<!DOCTYPE html>
-<html>
-<head>
-<title>Welcome to my Home Page</title>
-</head>
-<body>
-<h1>Welcome!</h1>
-<p>This is my website! Hi!</p>
-</body>
-</html>
-```
-
-Notice how the result of converting `index.md` into HTML is inserted into the template where `{{content}}` was, and the value of `title` from `index.md.meta` is inserted where `{{metadata.title}}`. While `index.md` inherited the top-level `title` metadata from the top `.meta`, its own `index.md.meta` file overrode it. Neat!
-
-The `publish` directory is ready to be serverd by a small HTTP server, placed in a web content directory, or whatever. We'll discuss that in a future section about hosting your Heckweasel site.
-
-## Getting (very slightly) more advanced with Heckweasel
-
-Now that we see how a project and its parts fit together we can make our little website slightly more interesting. 
-
-### Styling your Web Site
-
-As we alluded to above, the templates are where style information generally lives. 
-
-

docs/patterns.md

@@ -1,5 +0,0 @@
-# Patterns for Site Design #
-
-These are some simple patterns for things commonly needed in websites of various kinds.
-
-##

docs/quickstart.md

@@ -1,10 +0,0 @@
-# I am just writing a simple site with a couple of pages
-
-
-# I am interested in the technicalities of template development
-
-
-# I am interested in the technicalities of deployment
-
-
-

docs/templatefunctions.md

@@ -1,113 +0,0 @@
-# Template Functions #
-
-These are functions exposed to the templates which perform various useful actions for the site designer.
-
-## get_file_list ##
-
-Return a list of file names based on a wildcard glob, matched against the root of the project.
-
-Prototype: `get_file_list(file_glob, sort_order, reverse, limit) -> [files]`
-
-Arguments:
-* file_glob: A standard file glob, for example `*.txt` matches all files that end in `.txt` in the root of the project. (default: `*`)
-* sort_order: A string of either `file_path`, `file_name`, `ctime`, `mtime`, `size` and `ext` (default: `ctime`)
-* reverse: whether the sort is reversed (default: False)
-* limit: The number of entries to return from the top of the list, 0 for unlimited (default: `0`)
-
-Returns:
-* A list of file names.
-
-## get_file_name ##
-
-Return the filename that will result from processing the specified file based on the processors that it will be passed through.
-
-Prototype: `get_file_name(file) -> outfile`
-
-Arguments:
-* file: The name of a file, with path, from root.
-
-Returns:
-* outfile: The name of the file, with path, that will result from processing.
-
-## get_file_content ##
-
-Return the rendered content of specified file. Caution: Can result in infinite loops if two templates include each other.
-
-Prototype: `get_file_content(file) -> content`
-
-Arguments:
-* file: The name of the input file, with path, from root.
-
-Returns:
-* content: the contents that result from passing the specified file through its processors.
-
-## get_raw ##
-
-Return the raw contents of a source file. It is specifically not passed through any processing.
-
-Prototype: `get_raw(file) -> content`
-
-Arguments:
-* file: The name of the input file, with path, from root.
-
-Returns:
-* content: the raw contents of the input file
-
-## get_file_metadata ##
-
-Return the metadata tree associated with a particular file.
-
-Prototype: `get_file_metadata(file) -> metadata`
-
-Arguments:
-* file: the name of an input file, with path, from root
-
-Returns:
-* metadata: A dictionary of metadata loaded from the file tree.
-
-## get_time_iso8601 ##
-
-Return the date/time stamp in ISO 8601 format for a given time_t timestamp for UTC.
-
-Prototype: `get_time_iso8601(timestamp) -> timestamp`
-
-Arguments:
-* timestamp: A time_t integer or float, in seconds since Jan 1 1970.
-
-Returns:
-* timestamp: A string in ISO8601 format of the date and timestamp, in the UTC timezone.
-
-## get_date_iso8601 ##
-
-Return the date stamp in ISO 8601 format for a given time_t timestamp for UTC.
-
-Prototype: `get_date_iso8601(timestamp) -> timestamp`
-
-Arguments:
-* timestamp: A time_t integer or float, in seconds since Jan 1 1970.
-
-Returns:
-* timestamp: A string in ISO8601 format of the date stamp, in the UTC timezone.
-
-## pygments_get_css ##
-
-Return a blob of CSS produced from Pygments for a given `style`.
-
-Prototype: `pygments_get_css(style) -> css`
-
-Arguments:
-* style (optional): A style identifier for the Pygments' HTMLFormatter.
-
-Returns:
-* css: A string of styles as returned by Pygments' HTMLFormatter.
-
-## pygments_markup_contents_html ##
-
-Format a code fragment with Pygments
-
-Prototype: `pygments_markup_contents_html(input, filetype, style) -> html`
-
-Arguments:
-* input: A string containing the code to format (either literal, or imported with get_raw()).
-* filetype: A string describing which lexer to use.
-* style (optional) A style identifier for Pygments' HTMLFormatter.

examples/pixywerk.com/Makefile

@@ -0,0 +1,2 @@
+build: src/templates/* src/* src/images/* src/posts/*
+	python -mpixywerk2 src publish

examples/pixywerk.com/README.md

@@ -0,0 +1,4 @@
+# Pixywerk.com Example #
+
+This is an example blog system with the features most blogs would have (posts, tag cloud, atom/rss feeds,
+index with images).

examples/pixywerk.com/publish/css/main.css

@@ -0,0 +1,3 @@
+
+
+body { margin: 10% 10% 0 10% }

examples/pixywerk.com/publish/index.html

@@ -0,0 +1,13 @@
+<html>
+  <head>
+    <title></title>
+    <link rel="stylesheet" type="text/css" href="css/main.css">
+  </head>
+  <body>
+    <p>This is my index!!</p>
+
+for i in posts[:5]:
+
+get metadata, fill in post image/text summary with link
+  </body>
+</html>

examples/pixywerk.com/publish/posts/2019-04-15.html

@@ -0,0 +1,18 @@
+<html>
+  <head>
+    <title>My first post</title>
+    <link rel="stylesheet" type="text/css" href="css/main.css">
+  </head>
+  <body>
+    <img src="../images/20190415-0.jpg" class="featured">
+<div class="byline">
+  <p>Author: Cas Rusnov</p>
+  <p>Published: 2019-04-16T01:42:27.156392+00:00
+    
+  </p>
+</div>
+<p>This is an example post!</p>
+<p>yo fresh</p>
+<p>There are many posts like it but this one is mine.</p>
+  </body>
+</html>

examples/pixywerk.com/publish/templates/default.jinja2

@@ -0,0 +1,9 @@
+<html>
+  <head>
+    <title>{{ metadata.title }}</title>
+    <link rel="stylesheet" type="text/css" href="css/main.css">
+  </head>
+  <body>
+    {{ content }}
+  </body>
+</html>

examples/pixywerk.com/src/.meta

@@ -0,0 +1,6 @@
+{
+"author": "Cas Rusnov",
+"author_email": "rusnovn@gmail.com",
+"uuid-oid-root": "pixywerk.com/",
+"site_root": "https://pixywerk.com/"
+}

examples/pixywerk.com/src/css/main.css

@@ -0,0 +1,3 @@
+
+
+body { margin: 10% 10% 0 10% }

examples/pixywerk.com/src/index.thtml

@@ -0,0 +1,5 @@
+<p>This is my index!!</p>
+
+for i in posts[:5]:
+
+get metadata, fill in post image/text summary with link

examples/pixywerk.com/src/posts/2019-04-15.thtml

@@ -0,0 +1,12 @@
+<img src="{{ metadata.featured }}" class="featured">
+<div class="byline">
+  <p>Author: {{ metadata.author }}</p>
+  <p>Published: {{ get_time_iso8601(metadata.stat.ctime) }}
+    {% if metadata.stat.mtime-metadata.stat.ctime > 512 %}
+    Updated: {{ get_time_iso8601(metadata.stat.mtime) }}
+    {% endif %}
+  </p>
+</div>
+<p>This is an example post!</p>
+<p>yo fresh</p>
+<p>There are many posts like it but this one is mine.</p>

examples/pixywerk.com/src/posts/2019-04-15.thtml.meta

@@ -0,0 +1,4 @@
+{
+"title":"My first post",
+"featured":"../images/20190415-0.jpg"
+}

examples/pixywerk.com/src/templates/default.jinja2

@@ -0,0 +1,9 @@
+<html>
+  <head>
+    <title>{{ metadata.title }}</title>
+    <link rel="stylesheet" type="text/css" href="css/main.css">
+  </head>
+  <body>
+    {{ content }}
+  </body>
+</html>

heckweasel/__init__.py

@@ -1,14 +0,0 @@
-"""
-HeckWeasel: Metadata based static site compiler.
-"""
-__version__ = '0.7.1'
-__copyright__ = "©2023-2024 Aldercone Studio Collective"
-
-from . import metadata
-from . import processors
-from . import __main__
-from . import processchain
-from . import processors
-from . import template_tools
-from . import pygments
-from . import utils

heckweasel/__main__.py

@@ -1,283 +0,0 @@
-"""
-HeckWeasel command line interface.
-
-Performs compilation step given an input directory. See --help for more information.
-
-"""
-# iterate source tree
-#   create directors in target tree
-#   for each item:
-#    run processor(s) on item, each processor could be in a chain or a branch
-#    Processors also provide filename munging
-#    output target based on processor output
-
-import argparse
-import logging
-import os
-import shutil
-import sys
-import time
-
-import jinja2.exceptions
-
-from pathlib import Path
-from typing import Dict, List, cast, Union
-
-import tqdm
-
-from .metadata import MetaTree, gather_all_metadata, MetaDb
-from .processchain import ProcessorChains
-from .processors.processors import PassthroughException, NoOutputException
-from .pygments import pygments_get_css, pygments_markup_contents_html
-from .template_tools import (
-    date_iso8601,
-    file_content,
-    file_list,
-    file_list_hier,
-    file_json,
-    file_heck,
-    file_metadata,
-    file_name,
-    file_raw,
-    time_iso8601,
-    containsone,
-    sort_keys,
-)
-from .utils import deep_merge_dicts
-from .__init__ import __version__, __copyright__
-
-logger = logging.getLogger('heckweasel')
-
-logo = f"""
-Aldercone Studio Collective
- _           _                            _
-| |_  ___ __| |____ __ _____ __ _ ___ ___| |
-| ' \/ -_) _| / /\ V  V / -_) _` (_-</ -_) |
-|_||_\___\__|_\_\ \_/\_/\___\__,_/__/\___|_|
-{__version__}
-"""
-
-class TqdmLoggingHandler(logging.Handler):
-    """
-    A simple logging wrapper that won't clobber TQDM's progress bar.
-    """
-    def __init__(self, level=logging.NOTSET):
-        super().__init__(level)
-
-    def emit(self, record):
-        try:
-            msg = self.format(record)
-            tqdm.tqdm.write(msg)
-            self.flush()
-        except Exception:
-            self.handleError(record)
-
-def setup_logging(verbose:bool=False, quiet:bool=False, logfile:Union[Path, str, None]=None) -> None:
-    """
-    Configure logging based on some flags.
-    """
-    # Setup Tqdm handler
-    logger.setLevel(logging.DEBUG)
-    h = TqdmLoggingHandler()
-    if verbose:
-        f = logging.Formatter('%(asctime)s %(module)-12s %(levelname)-8s %(message)s')
-        h.setLevel(logging.DEBUG)
-        h.setFormatter(f)
-    elif quiet:
-        f = logging.Formatter('%(levelname)-8s %(message)s')
-        h.setLevel(logging.CRITICAL)
-        h.setFormatter(f)
-    else:
-        f = logging.Formatter('%(levelname)-8s %(message)s')
-        h.setLevel(logging.INFO)
-        h.setFormatter(f)
-    logger.addHandler(h)
-
-    # setup logfile if specified
-    if logfile:
-        lf = logging.FileHandler(logfile)
-        lf.setLevel(logging.DEBUG)
-        lf.setFormatter(logging.Formatter('%(asctime)s %(module)-12s %(levelname)-8s %(message)s'))
-        logger.addHandler(lf)
-
-def parse_var(varspec: str) -> List:
-    if (not ('=' in varspec)):
-        return [varspec, True]
-    return list(varspec.split('=', 2))
-
-
-def get_args(args: List[str]) -> argparse.Namespace:
-    parser = argparse.ArgumentParser("Compile a Heckweasel directory into an output directory.")
-
-    parser.add_argument("root", help="The root of the heckweasel directory to process.")
-    parser.add_argument("output", help="The output directory to export post-compiled files to.")
-
-    parser.add_argument(
-        "-c", "--clean", help="Remove the target tree before proceeding (by renaming to .bak).", action="store_true"
-    )
-    parser.add_argument("-s", "--safe", help="Abort if the target directory already exists.", action="store_true")
-    parser.add_argument("-f", "--follow-links", help="Follow symbolic links in the input tree.", action="store_true")
-    parser.add_argument("-t", "--template", help="The template directory (default: root/templates)", default=None)
-    parser.add_argument("-d", "--dry-run", help="Perform a dry-run.", action="store_true")
-    parser.add_argument("-v", "--verbose", help="Output verbosely.", action="store_true")
-    parser.add_argument("--debug", help="output extra debug info.", action="store_true")
-    parser.add_argument("--processors", help="Specify a path to a processor configuration file.", default=None)
-    parser.add_argument(
-        "-D", "--define", help="Add a variable to the metadata.", nargs="+", action="extend", type=parse_var)
-    result = parser.parse_args(args)
-    # validate arguments
-    if not os.path.isdir(result.root):
-        raise FileNotFoundError("can't find root folder {}".format(result.root))
-
-    if not result.template:
-        result.template = os.path.join(result.root, "templates")
-        result.excludes = [result.template]
-
-    return result
-
-
-def main() -> int:
-    print(logo)
-
-    try:
-        args = get_args(sys.argv[1:])
-    except FileNotFoundError as ex:
-        logger.info("error finding arguments: {}".format(ex))
-        return 1
-    setup_logging(args.verbose)
-    if os.path.exists(args.output) and args.clean:
-        bak = "{}.bak-{}".format(args.output, int(time.time()))
-        logger.info("cleaning target {} -> {}".format(args.output, bak))
-        os.rename(args.output, bak)
-
-    process_chains = ProcessorChains(args.processors)
-
-    default_metadata = {
-        "templates": args.template,
-        "template": "default.jinja2",
-        "dir-template": "default-dir.jinja2",
-        "filters": {},
-        "build-time": time.time(),
-        "uuid-oid-root": "heckweasel",
-        "summary": "",
-        "description": "",
-        "author": "",
-        "author_email": "",
-    }
-
-    if args.define:
-        for var in args.define:
-            default_metadata[var[0]] = var[1]
-
-
-    meta_tree = MetaTree(args.root, default_metadata)
-    file_list_cache = cast(Dict, {})
-    file_cont_cache = cast(Dict, {})
-    file_name_cache = cast(Dict, {})
-    file_raw_cache = cast(Dict, {})
-    flist = file_list(args.root, file_list_cache)
-
-    default_metadata["globals"] = {
-        "get_file_list": flist,
-        "get_hier": file_list_hier(args.root, flist),
-        "get_file_name": file_name(args.root, meta_tree, process_chains, file_name_cache),
-        "get_file_content": file_content(args.root, meta_tree, process_chains, file_cont_cache),
-        "get_json": file_json(args.root),
-        "get_heck": file_heck(args.root),
-        "get_raw": file_raw(args.root, file_raw_cache),
-        "get_file_metadata": file_metadata(meta_tree),
-        "get_time_iso8601": time_iso8601("UTC"),
-        "get_date_iso8601": date_iso8601("UTC"),
-        "pygments_get_css": pygments_get_css,
-        "pygments_markup_contents_html": pygments_markup_contents_html,
-        "merge_dicts": deep_merge_dicts,
-        "containsone": containsone,
-        'sort_keys': sort_keys,
-    }
-
-    # fixme add no-progress option for loop just to be the files
-
-
-    md = {}
-    haderrors = False
-    logger.info("Gathering all metadata")
-    for meta, item, error in tqdm.tqdm(gather_all_metadata(meta_tree,
-                                                           args.root,
-                                                           args.follow_links,
-                                                           [lambda x: x.endswith('.meta'), lambda x: x.endswith('~')]),
-                                       desc="Gathering metadata",
-                                       unit="files",
-                                       dynamic_ncols=True,
-                                       leave=False):
-        if (args.debug):
-            print(meta, item, error)
-        haderrors = error or haderrors
-        if args.verbose:
-            logger.debug(f"[dbg] {item} = {meta}")
-        if item in md:
-            logger.error("[!] multiple meta? ",  item)
-        md[item] = meta
-    mdb = MetaDb(md)
-
-    # technically metatree has all the md in its cache, but we also have md in a dictionary so who's to say. I guess
-    # we should make a separate object that lets you query md.
-
-    logger.info("Building Webbed Site")
-    for root, _, files in os.walk(args.root, followlinks=args.follow_links):
-        workroot = os.path.relpath(root, args.root)
-        if workroot == ".":
-            workroot = ""
-        target_dir = os.path.join(args.output, workroot)
-        logger.info("[D] Make directory -> {}".format(target_dir))
-        if not args.dry_run:
-            try:
-                os.mkdir(target_dir)
-            except FileExistsError:
-                if args.safe:
-                    logger.info("[A] Error, target directory exists and we are in safe mode, aborting")
-                    return 1
-        for f in tqdm.tqdm(files, desc="Building webbed site", unit="files", dynamic_ncols=True, leave=False):
-            # fixme global generic filters
-            try:
-                if f.endswith(".meta") or f.endswith("~"):
-                    continue
-                meta = md[os.path.join(workroot, f)]
-                meta['mdb'] = mdb
-                chain = process_chains.get_chain_for_filename(os.path.join(root, f), ctx=meta)
-                if args.verbose:
-                    logger.debug(f"metadata: {meta}")
-                logger.info("[P] Processing {} -> chains: {} -> output: {}".format(os.path.join(root, f), repr(chain), os.path.join(target_dir, chain.output_filename)))
-                if not args.dry_run:
-                    try:
-                        # normal output
-                        # FIXME support binary streams
-                        collected_output = [line for line in chain.output]
-                        with open(os.path.join(target_dir, chain.output_filename), "w") as outfile:
-                            outfile.writelines(collected_output)
-                    except PassthroughException:
-                        # write output from input
-                        shutil.copyfile(os.path.join(root, f), os.path.join(target_dir, chain.output_filename))
-                    except NoOutputException:
-                        logger.warn("[S] No content or output prevented {}".format(os.path.join(root, f), os.path.join(target_dir, chain.output_filename)))
-                        # don't write anyp output
-                        pass
-            except jinja2.exceptions.TemplateSyntaxError as inst:
-                logger.error(f"[!][S] Template error processing {f} Error was: {inst.filename}:{inst.lineno} {inst.message} (skipped)")
-                haderrors = True
-            except BaseException as inst:
-                # fixme optionally exit on error?
-                # print(inst.with_traceback())
-                # print(meta)
-                logger.error(f"[!][S] General error processing {f} Error was: {inst} (skipped)")
-                haderrors = True
-
-    if haderrors:
-        logger.error("One or more errors in processing.")
-        return 1
-    return 0
-
-def do_main():
-    sys.exit(main())
-
-if __name__ == "__main__":
-    do_main()

heckweasel/metadata.py

@@ -1,331 +0,0 @@
-"""Constructs andflag tree-like object containing the metadata for andflag given path, and caches said metadata."""
-
-import fnmatch
-import logging
-import mimetypes
-import os
-import uuid
-from typing import Any, Dict, List, Optional, Tuple, Union, cast, Callable, Iterable
-import yaml
-import copy
-
-import jstyleson
-
-import heckformat.parse
-
-from .utils import guess_mime, iterable
-
-# setup mimetypes with some extra ones
-mimetypes.init()
-mimetypes.add_type("text/html", "thtml")
-mimetypes.add_type("text/html", "cont")
-mimetypes.add_type("text/x-heckformat", "heck")
-
-logger = logging.getLogger(__name__)
-
-
-class MetaCacheMiss(Exception):
-    """Raised on cache miss."""
-
-class MetaLoadError(Exception):
-    "Raised when metadata fails to load."
-
-class MetaCache:
-    """This class provides an in-memory cache for metadata tree."""
-
-    def __init__(self, max_age: float = 200.0):
-        """Initialize the cache.
-
-        Arguments:
-            max_age (int): the number of seconds to age-out cache items
-
-        """
-        self._max_age = max_age
-        self._cache: Dict[str, Tuple[float, Any]] = {}
-
-    def get(self, key: str, new_time_stamp: float) -> Any:
-        """Get an item from the cache.
-
-        Arguments:
-            key (str): the cache key to retieve
-            new_time_stamp (int): The time to use to compare the stored time with
-
-        Returns:
-            :obj:misc: The previously stored value.
-
-        Raises:
-            MetaCacheMiss: on missing key, or on aged out
-
-        """
-        if (key not in self._cache):
-            raise MetaCacheMiss("no item for key {}".format(key))
-
-        if ((self._cache[key][0] + self._max_age) <= new_time_stamp):
-            return self._cache[key][1]
-
-        raise MetaCacheMiss("cache expired for key {}".format(key))
-
-    def put(self, key: str, value: Union[Dict, List, int, str, object], time_stamp: float) -> None:
-        """Put an item into the cache.
-
-        Arguments:
-            key (str): the key to store the cache item under
-            value (:obj:misc): the value to store
-            time_stamp (float): the time stamp to store the item under
-
-        """
-        self._cache[key] = (time_stamp, value)
-
-
-class MetaTree:
-    """This provides an interface to loading and caching tree metadata for andflag given directory tree."""
-
-    def __init__(self, root: str, default_metadata: Optional[Dict] = None):
-        """Initialize the metadata tree object.
-        Arguments:
-            root (str): The path to the root of the file tree to operate on.
-            default_metadata (dict, optional): The default metadata to apply to the tree
-
-        """
-        self._cache = MetaCache()
-        if (default_metadata is None):
-            default_metadata = {}
-        self._default_metadata = default_metadata
-        if (root[-1] != "/"):
-            root += "/"
-        self._root = root
-
-    def _get_cache_key(self, fullpath: str):
-        cachekey = fullpath + '.meta'
-        if fullpath.endswith(".heck"):
-            cachekey = fullpath
-        elif os.path.isdir(fullpath):
-            cachekey = os.path.join(fullpath, ".meta")
-            if (not os.path.exists(cachekey)):
-                cachekey = os.path.join(fullpath, ".heck")
-
-        return cachekey
-
-    def _load_metadata(self, cachekey: str) -> Dict:
-        meta = {}
-        try:
-            with open(cachekey, "r") as inf:
-                if cachekey.endswith(".heck"):
-                    with open(cachekey) as cachefile:
-                        h = heckformat.parse.load(cachefile)
-                        meta = h.flatten_replace()
-                else:
-                    try:
-                        # try json load
-                        meta = jstyleson.load(inf)
-                    except jstyleson.JSONDecodeError as exc:
-                        # try yaml load
-                        try:
-                            meta = yaml.load(inf)
-                        except yaml.parser.ParserError as exc2:
-                            # else either the yaml or json has an error
-                            me = MetaLoadError()
-                            exc2.__context__ = exc
-        except BaseException as inst:
-            logger.error(f"Can'tok load any metadata for key {cachekey}: {inst}")
-
-        return meta
-
-    def get_metadata(self, rel_path: str) -> Dict:
-        """Retrieve the metadata for andflag given path
-
-        The general procedure is to iterate the tree, at each level
-        load .meta (JSON formatted dictionary) for that level, and
-        then finally load the path.meta, and merge these dictionaries
-        in descendant order.
-
-        Arguments:
-            rel_path (str): The path to retrieve the metadata for (relative to root)
-
-        Returns:
-            dict: A dictionary of metadata for that path tree.
-
-        """
-        metablob = dict(self._default_metadata)
-        # iterate path components from root to target path
-        comps = [self._root] + rel_path.split("/")
-        fullpath = ""
-        ospath = os.path.join(self._root, rel_path)
-        for pth in comps:
-            fullpath = os.path.join(fullpath, pth)
-            st = os.stat(fullpath)
-
-            cachekey = self._get_cache_key(fullpath)
-
-            meta = cast(Dict, {})
-
-            try:
-                st_meta = os.stat(cachekey)
-                meta = self._cache.get(cachekey, st_meta.st_mtime)
-            except FileNotFoundError:
-                st_meta = None  # type: ignore
-            except MetaCacheMiss:
-                meta = {}
-
-            # if we didn'tok get any meta from the cache, but the metafile exists, try loading it
-            if ((not meta) and st_meta):
-                meta = self._load_metadata(cachekey)
-                self._cache.put(cachekey, meta, st_meta.st_mtime)
-
-            # add whatever is in the metablob as 'wildcard_metadata' to the metadata if the filename
-            # matches the wildcards
-            if ((fullpath == ospath) and ("wildcard_metadata" in metablob)):
-                for wild in metablob["wildcard_metadata"]:
-                    if fnmatch.fnmatch(pth, wild[0]):
-                        metablob.update(wild[1])
-
-            metablob.update(meta)
-
-        ### fill in all objective metadata
-        # containing directory and filename
-        metablob["dir"], metablob["fileName"] = os.path.split(rel_path)
-        # deprecated
-        metablob["dir"], metablob["file_name"] = os.path.split(rel_path)
-        # path within the source tree
-        metablob["filePath"] = rel_path
-        # deprecated
-        metablob["file_path"] = rel_path
-        # the path relative to the output tree
-        metablob["relpath"] = os.path.relpath("/", "/" + metablob["dir"])
-        # the UUID for this file
-        metablob["uuid"] = uuid.uuid3(uuid.NAMESPACE_OID, metablob["uuid-oid-root"] + ospath)
-        # the pre-split components of the full path
-        metablob["osPath"], _ = os.path.split(fullpath)
-        # deprecated
-        metablob["os-path"], _ = os.path.split(fullpath)
-        # the mime type we guessed for this file
-        metablob["guessedType"] = guess_mime(ospath)
-        # deprecated
-        metablob["guessed-type"] = guess_mime(ospath)
-        # if the mime-type isn'tok overriden in the explicit metadata, we make it equal to the guessed type
-        if "mime-type" not in metablob:
-            metablob["mime-type"] = metablob["guessed-type"]
-        # the `stat` components
-        metablob["stat"] = {}
-        for stk in ("st_mtime", "st_ctime", "st_atime", "st_mode", "st_size", "st_ino"):
-            metablob["stat"][stk.replace("st_", "")] = getattr(st, stk)
-
-        # return final dict
-        return metablob
-
-
-class MetaDb:
-    def __init__(self, data_dict: Optional[dict] = None):
-        if data_dict is None:
-            self.data = {}
-        else:
-            self.data = copy.deepcopy(data_dict)
-        self.indices = {}
-        self.indexed = False
-        self.build_indices()
-
-    def build_indices(self) -> None:
-        # build indices? like we could iterate all the items in data_dict, for each key, make an index for that key,
-        # we tehn set indexed to true and then search will search using indexes
-        ...
-
-    def fetch(self, key: Any):
-        # just return the item keyed as key or None
-        if (key in self.data):
-            return self.data[key]
-        return None
-
-    def _get_matches(self, matchk: str, matchv: Optional[Any]) -> set:
-        # fixme this is where we can do index based optimizations
-        result = set()
-        for key, value in self.data.items():
-            if (value.get(matchk, None) == matchv) or (matchv == '*'):
-                result.add(key)
-        return result
-
-    def _get_contains(self, matchk: str, matchv: Iterable) -> set:
-        result = set()
-        matchv = set(matchv)
-        for key, value in self.data.items():
-            v = value.get(matchk, None)
-            if (not iterable(v)):
-                v = set([v])
-            else:
-                v = set(v)
-            # print('***  ', key, matchk, v, '->', matchv, result)
-            if ('*' in matchv) or (len(matchv & v) >= 1):
-                result.add(key)
-        return result
-
-    def search(self, **kwargs) -> List[dict]:
-        # always x (or, and) y and not z, and is default
-        vand = {x for x in self.data.keys()}
-        vor = set()
-        vnot = set()
-
-        for key, value in kwargs.items():
-            andflag = True
-            orflag = False
-            notflag = False
-            containsflag = False
-            if ('_' in key):
-                tokens = key.split('_')
-                key = tokens.pop()
-                for tok in tokens:
-                    match tok:
-                        case 'or':
-                            orflag = True
-                            andflag = False
-                        case 'and':
-                            orflag = False
-                            andflag = True
-                        case 'not':
-                            notflag = True
-                        case 'in':
-                            containsflag = True
-                        case _:
-                          logger.warn(f"unknown metadata search flag `{tok}`")
-            if containsflag:
-                # do a contains operation
-                if (not iterable(value)):
-                    value = [value]
-                res = self._get_contains(key, value)
-            else:
-                # do a simple match operation
-                res = self._get_matches(key, value)
-            if notflag:
-                vnot = vnot | res
-            elif andflag:
-                vand = vand & res
-            elif orflag:
-                vor = vor | res
-            # print('** ', andflag, orflag, notflag, containsflag, key, value, res)
-
-        rkeys = (vand | vor) - vnot;
-        return {rk: self.data[rk] for rk in rkeys}
-
-
-
-def gather_all_metadata(mt: MetaTree, toproot: str, follow_links: bool, filters: Optional[List[Callable]] = None) -> Tuple:
-    if filters is None:
-        filters = []
-    for root, _, files in os.walk(toproot, followlinks=follow_links):
-        workroot = os.path.relpath(root, toproot)
-        if workroot == ".":
-            workroot = ""
-        for f in files:
-            pth = os.path.join(workroot, f)
-            try:
-                # fixme global generic filters
-                if any([x(pth) for x in filters]):
-                    continue
-                metadata = mt.get_metadata(pth)
-                yield metadata, pth, False
-            except BaseException as inst:
-                logger.error(f"[S] Error loading metadata for {pth} Error was: {inst} (skipped)")
-                yield dict(), pth, True
-
-def ___test(toproot):
-    mt = MetaTree(toproot, {'uuid-oid-root':'___test'})
-    md = {x[1]: x[0] for x in gather_all_metadata(mt, toproot, False, [])}
-    return MetaDb(md)

heckweasel/processors/__init__.py

@@ -1,12 +0,0 @@
-# processors metadata here
-
-from . import jinja2_page_embed
-from . import jinja2
-from . import passthrough
-from . import process_heck
-from . import process_less
-from . import process_md
-from . import processors
-from . import process_pp
-from . import process_sass
-from . import process_styl

heckweasel/processors/process_heck.py

@@ -1,85 +0,0 @@
-"""Convert a HECKformat file to a markdown stream."""
-
-import io
-import os
-
-from typing import Dict, Iterable, Optional
-
-import heckformat.parse
-
-from .processors import Processor, NoOutputException
-
-
-class HECKformatProcessor(Processor):
-    """Convert a HECKformat file to a markdown stream."""
-
-    def filename(self, oldname: str, ctx: Optional[Dict] = None) -> str:
-        """Return the filename of the post-processed file.
-
-        Arguments:
-            oldname (str): the previous name for the file.
-            ctx (dict, optional): A context object generated from the processor configuration
-
-        Returns:
-            str: the new name for the file
-
-        """
-        if ctx and 'heck_filename' in ctx:
-            return ctx['heck_filename']
-        
-        return os.path.splitext(oldname)[0] + this.extension(oldname, ctx)
-
-    def mime_type(self, oldname: str, ctx: Optional[Dict] = None) -> str:
-        """Return the mimetype of the post-processed file.
-
-        Arguments:
-            oldname (str): the input filename
-            ctx (dict, optional): A context object generated from the processor configuration
-
-        Returns:
-            str: the new mimetype of the file after processing
-
-        """
-        if ctx and 'heck_mime' in ctx:
-            return ctx['heck_mime']
-        
-        return "text/x-markdown"
-
-    def extension(self, oldname: str, ctx: Optional[Dict] = None) -> str:
-        """Return the mimetype of the post-processed file.
-
-        Arguments:
-            oldname (str): the input filename
-            ctx (dict, optional): A context object generated from the processor configuration
-
-        Returns:
-            str: the new extension of the file after processing
-
-        """
-        if ctx and 'heck_extension' in ctx:
-            return ctx['heck_extension']
-        
-        return "md"
-
-    def process(self, input_file: Iterable, ctx: Optional[Dict] = None) -> Iterable:
-        """Return an iterable object of the post-processed file.
-
-        Arguments:
-            input_file (iterable): An input stream
-            ctx (dict, optional): A context object generated from the processor configuration
-
-        Returns:
-            iterable: The post-processed output stream
-        """
-
-        elm = heckformat.parse.load_heck(input_file).flatten_replace()
-        for key in elm:
-            if key.startswith(heckformat.parse.UNPARSED_MARKER):
-                # fixme later we should use the doclabel to choose which output processor somehow~
-                doclabel = key.split(' ')[-1]
-                # we'll just assume the first unparsed part of the document is the page
-                return elm[key]
-        # No documents in the input heck, we just prevent output
-        raise NoOutputException()
-
-processor = HECKformatProcessor  # pylint: disable=invalid-name

heckweasel/pygments.py

@@ -1,36 +0,0 @@
-"""Map Pygments into the Template API for inclusion in outputs."""
-from typing import Optional, cast
-
-import pygments
-import pygments.formatters
-import pygments.lexers
-import pygments.styles
-import pygments.util
-
-
-def pygments_markup_contents_html(input_text: str, file_type: str, style: Optional[str] = None) -> str:
-    """Format input string with Pygments and return HTML."""
-
-    if style is None:
-        style = "default"
-    pyst = pygments.styles.get_style_by_name(style)
-    formatter = pygments.formatters.get_formatter_by_name("html", style=pyst)
-    try:
-        lexer = pygments.lexers.get_lexer_for_filename(file_type)
-    except pygments.util.ClassNotFound:
-        try:
-            lexer = pygments.lexers.get_lexer_by_name(file_type)
-        except pygments.util.ClassNotFound:
-            lexer = pygments.lexers.get_lexer_by_mimetype(file_type)
-
-    return pygments.highlight(input_text, lexer, formatter)
-
-
-def pygments_get_css(style: Optional[str] = None) -> str:
-    """Return the CSS styles associated with a particular style definition."""
-
-    if style is None:
-        style = "default"
-    pyst = pygments.styles.get_style_by_name(style)
-    formatter = pygments.formatters.get_formatter_by_name("html", style=pyst)
-    return formatter.get_style_defs()

heckweasel/template_tools.py

@@ -1,204 +0,0 @@
-"""
-Provides various utility functions that are exposed to the templates.
-"""
-import copy
-import datetime
-import glob
-import itertools
-import os
-from typing import Callable, Dict, Iterable, List, Union, cast, Tuple
-
-import jstyleson
-
-import pytz
-
-from .metadata import MetaTree
-from .processchain import ProcessorChains
-from .utils import deep_merge_dicts
-
-
-def file_list(root: str, listcache: Dict) -> Callable:
-    """
-    Return a function (memoized for the cache and root directory) which returns a list of files matching glob and
-    sorted as required.
-    """
-    def get_file_list(
-            path_glob: Union[str, List[str], Tuple[str]],
-            *,
-            sort_order: str = "ctime",
-            reverse: bool = False,
-            limit: int = 0) -> Iterable:
-        stattable = cast(List, [])
-        if isinstance(path_glob, str):
-            path_glob = [path_glob]
-        for pglob in path_glob:
-            if pglob in listcache:
-                stattable.extend(listcache[pglob])
-            else:
-                for fil in glob.glob(os.path.join(root, pglob)):
-                    if os.path.isdir(fil):
-                        continue
-                    if fil.endswith(".meta") or fil.endswith("~"):
-                        continue
-                    st = os.stat(fil)
-                    stattable.append(
-                        {
-                            "file_path": os.path.relpath(fil, root),
-                            "file_name": os.path.split(fil)[-1],
-                            "mtime": st.st_mtime,
-                            "ctime": st.st_ctime,
-                            "size": st.st_size,
-                            "ext": os.path.splitext(fil)[1],
-                        }
-                    )
-                listcache[pglob] = stattable
-            ret = sorted(stattable, key=lambda x: x[sort_order], reverse=reverse)
-        if limit > 0:
-            return itertools.islice(ret, limit)
-        return ret
-
-    return get_file_list
-
-
-def file_list_hier(root: str, flist: Callable) -> Callable:
-    """Return a function which, given a directory, will walk the directory and return the files within
-    it that match the glob passed."""
-
-    def get_file_list_hier(path: str, glob: str, *, sort_order: str = "ctime", reverse: bool = False) -> Iterable:
-        output = []
-
-        for pth in os.walk(os.path.join(root, path)):
-            output.extend(
-                flist(
-                    os.path.join(os.path.relpath(os.path.realpath(pth[0]), root), glob),
-                    sort_order=sort_order,
-                    reverse=reverse,
-                )
-            )
-
-        return output
-
-    return get_file_list_hier
-
-
-def file_name(root: str, metatree: MetaTree, processor_chains: ProcessorChains, namecache: Dict) -> Callable:
-    """
-    Return a function (memoized for root directory, metatree and processor chains) which returns the output filename
-    given an input filename based on metadata and said processing chains.
-    """
-    def get_file_name(file_name: str) -> Dict:
-        if file_name in namecache:
-            return namecache[file_name]
-        metadata = metatree.get_metadata(file_name)
-        chain = processor_chains.get_chain_for_filename(os.path.join(root, file_name), ctx=metadata)
-        namecache[file_name] = chain.output_filename
-        return namecache[file_name]
-
-    return get_file_name
-
-
-def file_raw(root: str, contcache: Dict) -> Callable:
-    """
-    Return a function (memoizedfor the root directory) which returns the raw content of a file.
-    """
-    def get_raw(file_name: str) -> str:
-        if file_name in contcache:
-            return contcache[file_name]
-        with open(os.path.join(root, file_name), "r", encoding="utf-8") as f:
-            return f.read()
-
-    return get_raw
-
-
-def file_json(root: str) -> Callable:
-    """
-    Return a function (memoized for the root directory) which loads a file as json, merges it with an optional input dictionary
-    and returns.
-    """
-    def get_json(file_name: str, parent: Dict = None) -> Dict:
-        outd = {}
-        if parent is not None:
-            outd = copy.deepcopy(parent)
-
-        with open(os.path.join(root, file_name), "r", encoding="utf-8") as f:
-            return deep_merge_dicts(outd, jstyleson.load(f))
-
-    return get_json
-
-
-def file_heck(root: str) -> Callable:
-    """
-    Return a function (memoized for the root directory) which loads a file as HECKFormat, merges it with an optional input
-    dictionary, and returns.
-    """
-    def get_heck(file_name: str, parent: Dict = None) -> Dict:
-        outd = {}
-        if parent is not None:
-            outd = copy.deepcopy(parent)
-
-        with open(os.path.join(root, file_name), "r", encoding="utf-8") as f:
-            return deep_merge_dicts(outd, heckformat.parse.load(f).flatten_replace())
-
-    return get_heck
-
-
-def file_content(root: str, metatree: MetaTree, processor_chains: ProcessorChains, contcache: Dict) -> Callable:
-    """
-    Return a function (memoized for the root directory, metatree, and processor chains) which returns the post-processed
-    content of the input file.
-    """
-    def get_file_content(file_name: str) -> Iterable:
-        if file_name in contcache:
-            return contcache[file_name]
-        metadata = metatree.get_metadata(file_name)
-        chain = processor_chains.get_chain_for_filename(os.path.join(root, file_name), ctx=metadata)
-        contcache[file_name] = chain.output
-        return str(chain.output)
-
-    return get_file_content
-
-
-def file_metadata(metatree: MetaTree) -> Callable:
-    """Returns a function (memoized for a metatree) which returns the meta data for a given file."""
-    def get_file_metadata(file_name: str) -> Dict:
-        return metatree.get_metadata(file_name)
-
-    return get_file_metadata
-
-
-def containsone(needle: Iterable, haystack: Iterable):
-    """
-    Returns true if at least one of the contents of needle is in haystack.
-    """
-    for n in needle:
-        if n in haystack:
-            return True
-
-    return False
-
-def time_iso8601(timezone: str) -> Callable:
-    """Returns a function (memoized for a particular timezone) which formats a time as ISO8601 standard. """
-    tz = pytz.timezone(timezone)
-
-    def get_time_iso8601(time_t: Union[int, float]) -> str:
-        return datetime.datetime.fromtimestamp(time_t, tz).isoformat("T")
-
-    return get_time_iso8601
-
-
-def date_iso8601(timezone: str) -> Callable:
-    """Returns a function (memoized for a particular timezone) which formats a date as ISO8601 standard. """
-    tz = pytz.timezone(timezone)
-
-    def get_date_iso8601(time_t: Union[int, float]) -> str:
-        return datetime.datetime.fromtimestamp(time_t, tz).strftime("%Y-%m-%d")
-
-    return get_date_iso8601
-
-
-def sort_keys(ind: dict, field: str, inv: bool = False, defval = None) -> str:
-    """Return the keys from a dictionary after sorting on a field of the dictionary's values."""
-    def dvcmp(a):
-        return a[1].get(field, defval)
-
-    return [x[0] for x in sorted(list(ind.items()), key=dvcmp, reverse=inv)]

heckweasel/utils.py

@@ -1,80 +0,0 @@
-from typing import Dict, Optional
-import copy
-import mimetypes
-import os
-import collections.abc
-import types
-
-def merge_dicts(dict_a: Dict, dict_b: Dict) -> Dict:
-    """Merge two dictionaries (shallow).
-
-    Arguments:
-        dict_a (dict): The dictionary to use as the base.
-        dict_b (dict): The dictionary to update the values with.
-
-    Returns:
-        dict: A new merged dictionary.
-
-    """
-    dict_z = dict_a.copy()
-    dict_z.update(dict_b)
-    return dict_z
-
-
-def deep_merge_dicts(dict_a: Dict, dict_b: Dict, _path=None, cpy=False) -> Dict:
-    """Merge two dictionaries (deep).
-    https://stackoverflow.com/questions/7204805/how-to-merge-dictionaries-of-dictionaries/7205107#7205107
-
-    Arguments:
-        dict_a (dict): The dictionary to use as the base.
-        dict_b (dict): The dictionary to update the values with.
-        _path (list): internal use.
-
-    Returns:
-        dict: A new merged dictionary.
-
-    """
-    if cpy:
-        dict_a = copy.deepcopy(dict_a)
-    if _path is None:
-        _path = []
-    for key in dict_b:
-        if key in dict_a:
-            if isinstance(dict_a[key], dict) and isinstance(dict_b[key], dict):
-                deep_merge_dicts(dict_a[key], dict_b[key], _path + [str(key)])
-            elif dict_a[key] == dict_b[key]:
-                pass # same leaf value
-            else:
-                dict_a[key] = copy.deepcopy(dict_b[key])
-        else:
-            dict_a[key] = dict_b[key]
-    return dict_a
-
-
-def guess_mime(path: str) -> Optional[str]:
-    """Guess the mime type for a given path.
-
-    Arguments:
-        root (str): the root path of the file tree
-        path (str): the sub-path within the file tree
-
-    Returns:
-        str: the guessed mime-type
-
-    """
-    # if path.endswith('.heck'):
-    #     return "text/x-heckformat"
-
-    mtypes = mimetypes.guess_type(path)
-    ftype = None
-    if os.path.isdir(path):
-        ftype = "directory"
-    elif os.access(path, os.F_OK) and mtypes[0]:
-        ftype = mtypes[0]
-    else:
-        ftype = "application/octet-stream"
-    return ftype
-
-
-def iterable(var):
-    return not (isinstance(var, str) or not isinstance(var, collections.abc.Iterable))

importwp.py

@@ -1,162 +0,0 @@
-"""Convert a Wordpress XML dump into to a (mostly working) heckweasel tree."""
-
-import argparse
-import datetime
-import json
-import os
-import sys
-from urllib.parse import urlparse
-from xml.etree.ElementTree import ElementTree
-
-import requests
-
-FILE_PATTERN = "{postdate}-{postname}.thtml"
-
-
-def parse_args(args):
-    parser = argparse.ArgumentParser("importwp.py")
-
-    parser.add_argument("input", help="The input file.")
-    parser.add_argument("out_dir", help="Output root directory.", default='.')
-    parser.add_argument("--fetch-attachments", help="Fetch all attachments referred to in file.", action="store_true", dest='fetch_attachments')
-    parser.add_argument("--attachment-dir", help="Subdirectory to place attachments in.", default="attachments", dest='attachment_dir')
-    parser.add_argument("--post-dir", help="Subdirectory to place posts in.", default="posts", dest='post_dir')
-    parser.add_argument("--page-dir", help="Subdirectory to place pages in.", default="", dest='page_dir')
-
-    result = parser.parse_args(args)
-    result.post_dir = os.path.join(result.out_dir, result.post_dir)
-    result.page_dir = os.path.join(result.out_dir, result.page_dir)
-    result.attachment_dir = os.path.join(result.out_dir, result.attachment_dir)
-
-    return result
-
-
-def parse_input(xmlpath):
-    tree = ElementTree()
-
-    tree_root = tree.parse(source=xmlpath)
-    posts = {}
-    attachments = {}
-    pages = {}
-
-    for node in tree_root.find("channel"):
-        if node.tag == "item":
-            post_type = node.find("{http://wordpress.org/export/1.2/}post_type")
-            if post_type is not None:
-                status = node.find("{http://wordpress.org/export/1.2/}status")
-                if status is not None and status.text == "draft":
-                    continue
-                content = node.find("{http://purl.org/rss/1.0/modules/content/}encoded")
-                title = node.find("title")
-                pubdate = node.find("pubDate")
-                description = node.find("description")
-                post_name = node.find("{http://wordpress.org/export/1.2/}post_name")
-                categories = node.findall("category")
-                post_id = node.find("{http://wordpress.org/export/1.2/}post_id")
-                post_parent = node.find("{http://wordpress.org/export/1.2/}post_parent")
-                if post_type.text == "post":
-                    # found a post!
-                    posts[post_id.text] = {'content':content,
-                                           'title':title,
-                                           'pubdate':pubdate,
-                                           'description':description,
-                                           'post_name':post_name,
-                                           'categories':categories,
-                                           'post_parent':post_parent}
-                elif post_type.text == "attachment":
-                    # attachment
-                    att_url = node.find("{http://wordpress.org/export/1.2/}attachment_url")
-
-                    attachments[post_id.text] = {'content':content,
-                                                 'title':title,
-                                                 'pubdate':pubdate,
-                                                 'description':description,
-                                                 'post_name':post_name,
-                                                 'categories':categories,
-                                                 'post_parent':post_parent,
-                                                 'att_url':att_url,}
-                elif post_type.text == "page":
-                    pages[post_id.text] = {'content':content,
-                                           'title':title,
-                                           'pubdate':pubdate,
-                                           'description':description,
-                                           'post_name':post_name,
-                                           'categories':categories,
-                                           'post_parent':post_parent}
-
-    return posts, attachments, pages
-
-def fetch_attachment(attch, outdir):
-    url = attch['att_url'].text
-    p = urlparse(url)
-    filename = os.path.join(outdir, os.path.split(p.path)[-1])
-    print("fetching attachment",url,"->",filename)
-    r = requests.get(url)
-    with open(filename, 'wb') as outf:
-        outf.write(r.content)
-
-def save_cont(post, outdir):
-    dt = datetime.datetime.strptime(post['pubdate'].text, "%a,  %d %b %Y %H:%M:%S %z")
-    postdate = dt.strftime("%Y-%m-%d-%H%M%S")
-    filename = FILE_PATTERN.format(postdate=postdate, postname=post['post_name'].text)
-    print(post['title'].text, "->", filename)
-    with open(os.path.join(outdir, filename), "w") as outf:
-        outf.write(post['content'].text)
-        # handle attachments
-
-        tags = []
-        category = ""
-        for tg in post['categories']:
-            if "domain" in tg.attrib and tg.attrib["domain"] == "category":
-                category = tg.text
-            else:
-                tags.append(tg.text)
-
-    with open(os.path.join(outdir, filename + ".meta"), "w") as outf:
-        metadata = {
-            "title": post['title'].text,
-            "description": post['description'].text,
-            "post_time": dt.timestamp(),
-            "featured": "",
-            "tags": tags,
-            "category": category,
-        }
-        json.dump(metadata, outf)
-
-
-def main():
-    args = parse_args(sys.argv[1:])
-    try:
-        os.mkdir(args.out_dir)
-    except FileExistsError:
-        pass
-
-    try:
-        os.mkdir(args.page_dir)
-    except FileExistsError:
-        pass
-
-    try:
-        os.mkdir(args.post_dir)
-    except FileExistsError:
-        pass
-
-    if args.fetch_attachments:
-        try:
-            os.mkdir(args.attachment_dir)
-        except FileExistsError:
-            pass
-
-    posts, attachments, pages = parse_input(args.input)
-
-    if args.fetch_attachments:
-        [fetch_attachment(post, args.attachment_dir) for post in attachments.values()]
-
-    [save_cont(post, args.post_dir) for post in posts.values()]
-    [save_cont(page, args.page_dir) for page in pages.values()]
-
-    return 0
-
-
-if __name__ == "__main__":
-    sys.exit(main())

pixywerk2/__main__.py

@@ -0,0 +1,128 @@
+# iterate source tree
+#   create directors in target tree
+#   for each item:
+#    run processor(s) on item, each processor could be in a chain or a branch
+#    Processors also provide filename munging
+#    output target based on processor output
+
+import argparse
+import logging
+import os
+import shutil
+import sys
+import time
+
+from typing import Dict, List, cast
+
+from .processchain import ProcessorChains
+from .processors.processors import PassthroughException
+from .metadata import MetaTree
+from .template_tools import file_list, file_name, file_content, file_metadata, time_iso8601
+
+
+logger = logging.getLogger()
+
+
+def setup_logging(verbose: bool = False) -> None:
+    pass
+
+
+def get_args(args: List[str]) -> argparse.Namespace:
+    parser = argparse.ArgumentParser("Compile a Pixywerk directory into an output directory.")
+
+    parser.add_argument("root", help="The root of the pixywerk directory to process.")
+    parser.add_argument("output", help="The output directory to export post-compiled files to.")
+
+    parser.add_argument(
+        "-c", "--clean", help="Remove the target tree before proceeding (by renaming to .bak).", action="store_true"
+    )
+    parser.add_argument("-s", "--safe", help="Abort if the target directory already exists.", action="store_true")
+    parser.add_argument("-t", "--template", help="The template directory (default: root/templates)", default=None)
+    parser.add_argument("-d", "--dry-run", help="Perform a dry-run.", action="store_true")
+    parser.add_argument("-v", "--verbose", help="Output verbosely.", action="store_true")
+    parser.add_argument("--processors", help="Specify a path to a processor configuration file.", default=None)
+
+    result = parser.parse_args(args)
+
+    # validate arguments
+    if not os.path.isdir(result.root):
+        raise FileNotFoundError("can't find root folder {}".format(result.root))
+
+    if not result.template:
+        result.template = os.path.join(result.root, "templates")
+        result.excludes = [result.template]
+
+    return result
+
+
+def main() -> int:
+    try:
+        args = get_args(sys.argv[1:])
+    except FileNotFoundError as ex:
+        print("error finding arguments: {}".format(ex))
+        return 1
+    setup_logging(args.verbose)
+    if os.path.exists(args.output) and args.clean:
+        bak = "{}.bak-{}".format(args.output, int(time.time()))
+        print("cleaning target {} -> {}".format(args.output, bak))
+        os.rename(args.output, bak)
+
+    process_chains = ProcessorChains(args.processors)
+
+    default_metadata = {
+        "templates": args.template,
+        "template": "default.jinja2",
+        "dir-template": "default-dir.jinja2",
+        "filters": {},
+        "build-time": time.time(),
+        "uuid-oid-root": "pixywerk",
+        "summary": "",
+        "description": "",
+        "author": "",
+        "author_email": ""
+    }
+    meta_tree = MetaTree(args.root, default_metadata)
+    file_list_cache = cast(Dict, {})
+    file_cont_cache = cast(Dict, {})
+    file_name_cache = cast(Dict, {})
+    default_metadata["globals"] = {
+        "get_file_list": file_list(args.root, file_list_cache),
+        "get_file_name": file_name(args.root, meta_tree, process_chains, file_name_cache),
+        "get_file_content": file_content(args.root, meta_tree, process_chains, file_cont_cache),
+        "get_file_metadata": file_metadata(meta_tree),
+        "get_time_iso8601": time_iso8601("UTC"),
+    }
+
+    for root, _, files in os.walk(args.root):
+        workroot = os.path.relpath(root, args.root)
+        if workroot == ".":
+            workroot = ""
+        target_dir = os.path.join(args.output, workroot)
+        print("mkdir -> {}".format(target_dir))
+        if not args.dry_run:
+            try:
+                os.mkdir(target_dir)
+            except FileExistsError:
+                if args.safe:
+                    print("error, target directory exists, aborting")
+                    return 1
+        for f in files:
+            # fixme global generic filters
+            if f.endswith(".meta") or f.endswith("~"):
+                continue
+            metadata = meta_tree.get_metadata(os.path.join(workroot, f))
+            chain = process_chains.get_chain_for_filename(os.path.join(root, f), ctx=metadata)
+            print("process {} -> {}".format(os.path.join(root, f), os.path.join(target_dir, chain.output_filename)))
+            if not args.dry_run:
+                try:
+                    with open(os.path.join(target_dir, chain.output_filename), "w") as outfile:
+                        for line in chain.output:
+                            outfile.write(line)
+                except PassthroughException:
+                    shutil.copyfile(os.path.join(root, f), os.path.join(target_dir, chain.output_filename))
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

pixywerk2/defaults/chains.yaml

@@ -0,0 +1,104 @@
+# Default: output == input
+default:
+    extension: default
+    chain:
+        - passthrough
+
+# Any object that needs jinja scripts but no other explicit processing
+templatable:
+    extension: null
+    chain:
+        - jinja2
+
+# Markdown, BBCode and RST are first run through the templater, and then
+# they are processed into HTML, and finally embedded in a page template.
+markdown:
+    extension:
+        - md
+    chain:
+        - jinja2
+        - process_md
+        - jinja2_page_embed
+bbcode:
+    extension:
+        - bb
+        - pp
+    chain:
+        - jinja2
+        - process_pp
+        - jinja2_page_embed
+# FIXME implement RST processor
+# restructured:
+#     extension:
+#         - rst
+#     chain:
+#         - jinja2
+#         - process_rst
+#         - jinja2_page_embed
+
+# # JSON and YAML are split, passed through a pretty printer, and then output
+# FIXME implement split chain processor, implement processor arguments
+# json:
+#     extension:
+#         - json
+#     chain:
+#         - split (passthrough)
+#         - pp_json
+# yaml:
+#     extension:
+#         - yml
+#         - yaml
+#     chain:
+#         - split (passthrough)
+#         - pp_yaml
+
+# Template-html is first passed through the templater, and then embedded
+# in a page template
+template-html:
+    extension:
+        - thtml
+        - cont
+    chain:
+        - jinja2
+        - jinja2_page_embed
+
+# Smart CSS are simply converted to CSS.
+sass:
+    extension:
+        - sass
+        - scss
+    chain:
+        - process_sass
+less:
+    extension:
+        - less
+    chain:
+        - process_less
+
+stylus:
+    extension:
+        - styl
+    chain:
+        - process_styl
+
+# # Images are processed into thumbnails and sized in addition to being retained as their original
+# FIXME implement split chain processor, implement processor arguments,
+# image:
+#     extension:
+#         - jpg
+#         - jpeg
+#         - png
+#     chain:
+#         - split (image_bigthumb)
+#         - split (image_smallthumb)
+#         - passthrough
+
+# image_bigthumb:
+#     extension:
+#     chain:
+#         - smart_resize (big)
+
+# image_smallthumb:
+#     extension:
+#     chain:
+#         - smart_resize (small)

pixywerk2/metadata.py

@@ -0,0 +1,145 @@
+"""Constructs a tree-like object containing the metadata for a given path, and caches said metadata."""
+
+import logging
+import mimetypes
+import os
+import uuid
+
+from typing import Dict, Optional, Union, List, Tuple, Any, cast
+
+import jstyleson
+
+from .utils import guess_mime
+
+# setup mimetypes with some extra ones
+mimetypes.init()
+mimetypes.add_type("text/html", "thtml")
+mimetypes.add_type("text/html", "cont")
+
+logger = logging.getLogger(__name__)
+
+
+class MetaCacheMiss(Exception):
+    """Raised on cache miss."""
+
+
+class MetaCache:
+    """This class provides an in-memory cache for metadata tree."""
+
+    def __init__(self, max_age: float = 200.0):
+        """Initialize the cache.
+
+        Arguments:
+            max_age (int): the number of seconds to age-out cache items
+
+        """
+        self._max_age = max_age
+        self._cache: Dict[str, Tuple[float, Any]] = {}
+
+    def get(self, key: str, new_time_stamp: float) -> Any:
+        """Get an item from the cache.
+
+        Arguments:
+            key (str): the cache key to retieve
+            new_time_stamp (int): The time to use to compare the stored time with
+
+        Returns:
+            :obj:misc: The previously stored value.
+
+        Raises:
+            MetaCacheMiss: on missing key, or on aged out
+
+        """
+        if key not in self._cache:
+            raise MetaCacheMiss("no item for key {}".format(key))
+
+        if self._cache[key][0] + self._max_age <= new_time_stamp:
+            return self._cache[key][1]
+
+        raise MetaCacheMiss("cache expired for key {}".format(key))
+
+    def put(self, key: str, value: Union[Dict, List, int, str, object], time_stamp: float) -> None:
+        """Put an item into the cache.
+
+        Arguments:
+            key (str): the key to store the cache item under
+            value (:obj:misc): the value to store
+            time_stamp (float): the time stamp to store the item under
+
+        """
+        self._cache[key] = (time_stamp, value)
+
+
+class MetaTree:
+    """This provides an interface to loading and caching tree metadata for a given directory tree."""
+
+    def __init__(self, root: str, default_metadata: Optional[Dict] = None):
+        """Initialize the metadata tree object.
+
+        Arguments:
+            root (str): The path to the root of the file tree to operate on.
+            default_metadata (dict, optional): The default metadata to apply to the tree
+
+        """
+        self._cache = MetaCache()
+        if default_metadata is None:
+            default_metadata = {}
+        self._default_metadata = default_metadata
+        if root[-1] != "/":
+            root += "/"
+        self._root = root
+
+    def get_metadata(self, rel_path: str) -> Dict:
+        """Retrieve the metadata for a given path
+
+        The general procedure is to iterate the tree, at each level
+m        load .meta (JSON formatted dictionary) for that level, and
+        then finally load the path.meta, and merge these dictionaries
+        in descendant order.
+
+        Arguments:
+            rel_path (str): The path to retrieve the metadata for (relative to root)
+
+        Returns:
+            dict: A dictionary of metadata for that path tree.
+
+        """
+        metablob = dict(self._default_metadata)
+        # iterate path components from root to target path
+        comps = [self._root] + rel_path.split("/")
+        fullpath = ""
+        for pth in comps:
+            fullpath = os.path.join(fullpath, pth)
+            st = os.stat(fullpath)
+
+            cachekey = fullpath + ".meta"
+            meta = cast(Dict, {})
+            try:
+                st_meta = os.stat(cachekey)
+                meta = self._cache.get(cachekey, st_meta.st_mtime)
+            except FileNotFoundError:
+                st_meta = None  # type: ignore
+            except MetaCacheMiss:
+                meta = {}
+
+            if not meta and st_meta:
+                meta = jstyleson.load(open(cachekey, "r"))
+                self._cache.put(cachekey, meta, st_meta.st_mtime)
+
+            metablob.update(meta)
+
+        # return final dict
+        metablob["dir"], metablob["file_name"] = os.path.split(rel_path)
+        metablob["file_path"] = rel_path
+        metablob["uuid"] = uuid.uuid3(
+            uuid.NAMESPACE_OID, metablob["uuid-oid-root"] + os.path.join(self._root, rel_path)
+        )
+        metablob["os-path"], _ = os.path.split(fullpath)
+        metablob["guessed-type"] = guess_mime(os.path.join(self._root, rel_path))
+        if "mime-type" not in metablob:
+            metablob["mime-type"] = metablob["guessed-type"]
+        metablob["stat"] = {}
+        for stk in ("st_mtime", "st_ctime", "st_atime", "st_mode", "st_size", "st_ino"):
+            metablob["stat"][stk.replace("st_", "")] = getattr(st, stk)
+
+        return metablob

pixywerk2/processchain.py

@@ -3,32 +3,13 @@
 import os
 import os.path
 import random
-from typing import Any, Dict, Iterable, List, Optional, Type, cast
+
+from typing import List, Iterable, Optional, Any, Dict, Type, cast
 
 import yaml
 
 from .processors.processors import Processor
 
-PROCESS_CHAIN_DEFAULT = {
-    'default': {'extension': 'default',
-                'chain': ['passthrough']
-                },
-    'templatable': {'extension': None,
-                    'chain': ['jinja2']
-                    },
-    'tembed': {'extension': None,
-               'chain': ['jinja2', 'jinja2_page_embed']
-               },
-    'markdown': {'extension': ['md'],
-                 'chain': ['jinja2', 'process_md', 'jinja2_page_embed']},
-    'bbcode': {'extension': ['bb', 'pp'],
-               'chain': ['jinja2', 'process_pp', 'jinja2_page_embed']},
-    'template-html': {'extension': ['thtml', 'cont'],
-                      'chain': ['jinja2', 'jinja2_page_embed']},
-    'heckformat': {'extension': ['heck'],
-                   'chain': ['process_heck', 'jinja2', 'process_md', 'jinja2_page_embed']}
-}
-
 
 class ProcessorChain:
     """This implements a wrapper for an arbitrary set of processors and an associated file stream."""
@@ -110,9 +91,6 @@ class ProcessorChain:
             fname = processor.filename(fname, self._ctx)
         return fname
 
-    def __repr__(self) -> str:
-        return "[" + ",".join([x.__class__.__name__ for x in self._processors]) + "]"
-
 
 class ProcessorChains:
     """Load a configuration for processor chains, and provide ability to process the chains given a particular input
@@ -127,9 +105,9 @@ class ProcessorChains:
 
         """
         if config is None:  # pragma: no coverage
-            self.chainconfig = PROCESS_CHAIN_DEFAULT
-        else:
-            self.chainconfig = yaml.full_load(open(config, "r"))
+            config = os.path.join(os.path.dirname(__file__), "defaults", "chains.yaml")
+
+        self.chainconfig = yaml.load(open(config, "r"))
         self.extensionmap: Dict[str, Any] = {}
         self.processors: Dict[str, Type[Processor]] = {}
         for ch, conf in self.chainconfig.items():
@@ -166,10 +144,7 @@ class ProcessorChains:
                 ftype = "default"
 
         if ctx and "type" in ctx:
-            if isinstance(ctx["type"], str):
-                ftype = ctx["type"]
-            else:
-                ftype = ctx["type"][0]
+            ftype = ctx["type"]
         return self.get_chain_for_file(open(filename, "r"), ftype, filename, ctx)
 
     def get_chain_for_file(
@@ -196,12 +171,8 @@ class ProcessorChains:
         if not (bool(file_name)):
             file_name = hex(random.randint(0, 65536))
 
-        chainconfig =  self.chainconfig[file_type]["chain"]
-        if (ctx and 'process_chain' in ctx):
-            chainconfig = ctx['process_chain']
-
         return ProcessorChain(
-            [self.processors[x]() for x in chainconfig],
+            [self.processors[x]() for x in self.chainconfig[file_type]["chain"]],
             cast(str, file_name),
             file_obj,
             file_type,

pixywerk2/processors/__init__.py

@@ -0,0 +1 @@
+# processors metadata here

pixywerk2/processors/jinja2.py

@@ -1,6 +1,6 @@
 """Define a Jinja2 Processor which applies programmable templating to the input stream."""
 
-from typing import Dict, Iterable, Optional, cast
+from typing import Iterable, Optional, Dict, cast
 
 from jinja2 import Environment, FileSystemLoader
 
@@ -21,12 +21,12 @@ class Jinja2(PassThrough):
         Returns:
             iterable: The post-processed output stream
         """
-        if (ctx is None):
-            ctx = {}
-        template_env = Environment(loader=FileSystemLoader(ctx["templates"]), extensions=["jinja2.ext.do"])
+        ctx = cast(Dict, ctx)
+        template_env = Environment(loader=FileSystemLoader(ctx["templates"]))
         template_env.globals.update(ctx["globals"])
         template_env.filters.update(ctx["filters"])
         tmpl = template_env.from_string("".join([x for x in input_file]))
         return tmpl.render(metadata=ctx)
 
+
 processor = Jinja2

pixywerk2/processors/jinja2_page_embed.py

@@ -3,7 +3,8 @@
    the target template is rendered)."""
 
 import os
-from typing import Dict, Iterable, Optional, cast
+
+from typing import Iterable, Optional, Dict, cast
 
 from jinja2 import Environment, FileSystemLoader
 
@@ -24,7 +25,8 @@ class Jinja2PageEmbed(Processor):
             str: the new name for the file
 
         """
-        return os.path.splitext(oldname)[0] + "." + self.extension(oldname, ctx)
+
+        return os.path.splitext(oldname)[0] + ".html"
 
     def mime_type(self, oldname: str, ctx: Optional[Dict] = None) -> str:
         """Return the mimetype of the post-processed file.
@@ -37,7 +39,7 @@ class Jinja2PageEmbed(Processor):
             str: the new mimetype of the file after processing
 
         """
-        return ctx.get("mime", "text/html")
+        return "text/html"
 
     def process(self, input_file: Iterable, ctx: Optional[Dict] = None) -> Iterable:
         """Return an iterable object of the post-processed file.
@@ -50,15 +52,10 @@ class Jinja2PageEmbed(Processor):
             iterable: The post-processed output stream
         """
         ctx = cast(Dict, ctx)
-        template_env = Environment(loader=FileSystemLoader(ctx["templates"]), extensions=["jinja2.ext.do"])
+        template_env = Environment(loader=FileSystemLoader(ctx["templates"]))
         template_env.globals.update(ctx["globals"])
         template_env.filters.update(ctx["filters"])
-        if isinstance(ctx["template"], str):
-            tmpl = template_env.get_template(ctx["template"])
-        else:
-            # we've got a heck
-            tmpl = template_env.get_template(ctx["template"][0])
-        # print(tmpl)
+        tmpl = template_env.get_template(ctx["template"])
         content = "".join([x for x in input_file])
         return tmpl.render(content=content, metadata=ctx)
 
@@ -73,7 +70,7 @@ class Jinja2PageEmbed(Processor):
             str: the new extension of the file after processing
 
         """
-        return ctx.get("extension", "html")
+        return "html"
 
 
 processor = Jinja2PageEmbed

pixywerk2/processors/passthrough.py

@@ -1,10 +1,10 @@
 """Passthrough progcessor which takes input and returns it."""
 
 import os
-from typing import Dict, Iterable, Optional, cast
 
+from .processors import Processor, PassthroughException
 from ..utils import guess_mime
-from .processors import PassthroughException, Processor
+from typing import Iterable, Optional, Dict, cast
 
 
 class PassThrough(Processor):

pixywerk2/processors/process_md.py

@@ -2,7 +2,8 @@
 
 import io
 import os
-from typing import Dict, Iterable, Optional
+
+from typing import Iterable, Optional, Dict
 
 import markdown
 

pixywerk2/processors/processors.py

@@ -1,12 +1,11 @@
 import abc
-from typing import Dict, Iterable, Optional
+
+from typing import Iterable, Optional, Dict
 
 
 class PassthroughException(Exception):
     """Raised when the processor would like the file to pass through unchanged."""
 
-class NoOutputException(Exception):
-    """Raised when the processor would like no output to be written from the processing chain."""
 
 class ProcessorException(Exception):  # pragma: no cover
     """A base exception class to be used by processor objects."""
@@ -66,6 +65,3 @@ class Processor(abc.ABC):  # pragma: no cover
         Returns:
             iterable: The post-processed output stream
         """
-
-    def repr(self) -> str:
-        return self.__class__.__name__

pixywerk2/template_tools.py

@@ -0,0 +1,80 @@
+import datetime
+import glob
+import itertools
+import os
+import pytz
+from typing import Callable, Dict, List, Iterable, Union, cast
+
+from .metadata import MetaTree
+from .processchain import ProcessorChains
+
+
+def file_list(root: str, listcache: Dict) -> Callable:
+    def get_file_list(path_glob: str, *, sort_order: str = "ctime", reverse: bool = False, limit: int = 0) -> Iterable:
+        stattable = cast(List, [])
+        if path_glob in listcache:
+            stattable = listcache[path_glob]
+        else:
+            for fil in glob.glob(os.path.join(root, path_glob)):
+                if os.path.isdir(fil):
+                    continue
+                if fil.endswith(".meta") or fil.endswith("~"):
+                    continue
+                st = os.stat(fil)
+                stattable.append(
+                    {
+                        "file_path": os.path.relpath(fil, root),
+                        "file_name": os.path.split(fil)[-1],
+                        "mtime": st.st_mtime,
+                        "ctime": st.st_ctime,
+                        "size": st.st_size,
+                        "ext": os.path.splitext(fil)[1],
+                    }
+                )
+            listcache[path_glob] = stattable
+        ret = sorted(stattable, key=lambda x: x[sort_order], reverse=reverse)
+        if limit > 0:
+            return itertools.islice(ret, limit)
+        return ret
+
+    return get_file_list
+
+
+def file_name(root: str, metatree: MetaTree, processor_chains: ProcessorChains, namecache: Dict) -> Callable:
+    def get_file_name(file_name: str) -> Dict:
+        if file_name in namecache:
+            return namecache[file_name]
+        metadata = metatree.get_metadata(file_name)
+        chain = processor_chains.get_chain_for_filename(os.path.join(root, file_name), ctx=metadata)
+        namecache[file_name] = chain.output_filename
+        return namecache[file_name]
+
+    return get_file_name
+
+
+def file_content(root: str, metatree: MetaTree, processor_chains: ProcessorChains, contcache: Dict) -> Callable:
+    def get_file_content(file_name: str) -> Iterable:
+        if file_name in contcache:
+            return contcache[file_name]
+        metadata = metatree.get_metadata(file_name)
+        chain = processor_chains.get_chain_for_filename(os.path.join(root, file_name), ctx=metadata)
+        contcache[file_name] = chain.output
+        return chain.output
+
+    return get_file_content
+
+
+def file_metadata(metatree: MetaTree) -> Callable:
+    def get_file_metadata(file_name: str) -> Dict:
+        return metatree.get_metadata(file_name)
+
+    return get_file_metadata
+
+
+def time_iso8601(timezone: str) -> Callable:
+    tz = pytz.timezone(timezone)
+
+    def get_time_iso8601(time_t: Union[int, float]) -> str:
+        return datetime.datetime.fromtimestamp(time_t, tz).isoformat("T")
+
+    return get_time_iso8601

pixywerk2/utils.py

@@ -0,0 +1,42 @@
+import mimetypes
+import os
+
+from typing import Dict, Optional
+
+
+def merge_dicts(dict_a: Dict, dict_b: Dict) -> Dict:
+    """Merge two dictionaries.
+
+    Arguments:
+        dict_a (dict): The dictionary to use as the base.
+        dict_b (dict): The dictionary to update the values with.
+
+    Returns:
+        dict: A new merged dictionary.
+
+    """
+    dict_z = dict_a.copy()
+    dict_z.update(dict_b)
+    return dict_z
+
+
+def guess_mime(path: str) -> Optional[str]:
+    """Guess the mime type for a given path.
+
+    Arguments:
+        root (str): the root path of the file tree
+        path (str): the sub-path within the file tree
+
+    Returns:
+        str: the guessed mime-type
+
+    """
+    mtypes = mimetypes.guess_type(path)
+    ftype = None
+    if os.path.isdir(path):
+        ftype = "directory"
+    elif os.access(path, os.F_OK) and mtypes[0]:
+        ftype = mtypes[0]
+    else:
+        ftype = "application/octet-stream"
+    return ftype

pyproject.toml

@@ -1,21 +0,0 @@
-[build-system]
-requires      = ["pdm-backend"]
-build-backend = "pdm.backend"
-
-
-[project]
-name = "heckweasel"
-dynamic = ["version"]
-description = "A metadata based static site compiler with CMS-like features."
-authors = [{name = "Cassowary", email="cassowary@aldercone.studio"}]
-dependencies = ["yaml-1.3", "markdown", "jstyleson", "jinja2", "pygments", "heckformat"]
-requires-python = ">=3.8"
-readme = "README.md"
-license = {text = "LICENSE"}
-
-[tool.pdm.version]
-source = "file"
-path = "heckweasel/__init__.py"
-
-[project.scripts]
-heckweasel = "heckweasel.__main__:do_main"

setup.py

@@ -0,0 +1,59 @@
+"""Package configuration."""
+from setuptools import find_packages, setup
+
+LONG_DESCRIPTION = """Pixywerk 2 is a filesystem based static site generator."""
+
+INSTALL_REQUIRES = ["yaml-1.3", "markdown", "jstyleson", "jinja2"]
+
+# Extra dependencies
+EXTRAS_REQUIRE = {
+    # Test dependencies
+    "tests": [
+        "black",
+        "bandit>=1.1.0",
+        "flake8>=3.2.1",
+        "mypy>=0.470",
+        "prospector[with_everything]>=0.12.4",
+        "pytest-cov>=1.8.0",
+        "pytest-xdist>=1.15.0",
+        "pytest>=3.0.3",
+        "sphinx_rtd_theme>=0.1.6",
+        "sphinx-argparse>=0.1.15",
+        "Sphinx>=1.4.9",
+    ]
+}
+
+SETUP_REQUIRES = ["pytest-runner>=2.7.1", "setuptools_scm>=1.15.0"]
+setup(
+    author="Cassowary Rusnov",
+    author_email="rusnovn@gmail.com",
+    classifiers=[
+        "Development Status :: 1 - Pre-alpha",
+        "Environment :: Console",
+        "License :: OSI Approved :: MIT",
+        "Operating System :: POSIX :: Linux",
+        "Programming Language :: Python :: 3.6",
+        "Programming Language :: Python :: 3.7",
+        "Programming Language :: Python :: 3 :: Only",
+        "Topic :: Software Development :: Libraries :: Python Modules",
+    ],
+    description="A filesystem-based website generator / CMS",
+    # entry_points={
+    #    'console_scripts': [
+    #        'cookbook = spicerack.cookbook:main',
+    #    ],
+    # },
+    include_package_data=True,
+    extras_require=EXTRAS_REQUIRE,
+    install_requires=INSTALL_REQUIRES,
+    keywords=["cms", "website", "compiler"],
+    license="MIT",
+    long_description=LONG_DESCRIPTION,
+    name="pixywerk2",
+    packages=find_packages(exclude=["*.tests", "*.tests.*"]),
+    platforms=["GNU/Linux"],
+    setup_requires=SETUP_REQUIRES,
+    use_scm_version=True,
+    url="https://git.antpanethon.com/cas/pixywerk2",
+    zip_safe=False,
+)

tox.ini

@@ -1,5 +1,5 @@
 [tox]
-envlist=py{36,37,38,39}-{code-quality, unit} #, py37-sphinx
+envlist=py{36,37}-{code-quality, unit} #, py37-sphinx
 skipsdist = true
 
 [testenv]
@@ -7,18 +7,16 @@ setenv =
     LANG = en_US.UTF-8
 deps = .[tests]
 commands =
-	 unit: py.test --strict --cov-report=term-missing --cov=heckweasel heckweasel/tests/unit {posargs}
-	 code-quality: flake8 heckweasel
-	 code-quality: black -l 120 --check heckweasel
+	 unit: py.test --strict --cov-report=term-missing --cov=pixywerk2 pixywerk2/tests/unit {posargs}
+	 code-quality: flake8 pixywerk2
+	 code-quality: black -l 120 --check pixywerk2
 	 code-quality: - prospector -A
-	 code-quality: - mypy --ignore-missing-imports heckweasel
+	 code-quality: - mypy --ignore-missing-imports pixywerk2
 	 # sphinx: python setup.py build_sphinx -b html
    	 # sphinx: python setup.py build_sphinx -b man
 basepython =
     py36: python3.6
     py37: python3.7
-    py38: python3.8
-    py39: python3.9
 
 [flake8]
 max-line-length = 120