diff --git a/LICENSE.md b/LICENSE.md new file mode 100644 index 0000000..c617034 --- /dev/null +++ b/LICENSE.md @@ -0,0 +1,2 @@ +* TeX Live: +* LaTeX: diff --git a/Makefile b/Makefile index ad4cde3..a240862 100644 --- a/Makefile +++ b/Makefile @@ -28,7 +28,16 @@ example.pdf: STACK_NAME ?= latex-layer -build/output.yaml: template.yaml result/bin/x86_64-linux/latex +#result/texlive/bin/x86_64-linux/pdflatex: all + +build/layer.zip: result/texlive/bin/x86_64-linux/pdflatex build + # CloudFormation has trouble zipping texlive due to nested directory dept + # + # This is why we zip outside + + cd result && zip -ry $(PROJECT_ROOT)$@ * + +build/output.yaml: template.yaml build/layer.zip aws cloudformation package --template $< --s3-bucket $(DEPLOYMENT_BUCKET) --output-template-file $@ deploy: build/output.yaml diff --git a/Makefile_Latex b/Makefile_Latex index 360a421..ab5c41a 100644 --- a/Makefile_Latex +++ b/Makefile_Latex @@ -9,7 +9,7 @@ $(TEXLIVE_SOURCE): curl -LO http://mirror.ctan.org/systems/texlive/tlnet/$(TEXLIVE_SOURCE) $(MD5_pm): - yum install perl-Digest-MD5 -y + yum install -y perl-Digest-MD5 $(WGET): yum install -y wget diff --git a/README-SAR.md b/README-SAR.md new file mode 100644 index 0000000..bd5c64f --- /dev/null +++ b/README-SAR.md @@ -0,0 +1,20 @@ +# TeX Live (latex/pdflatex) for AWS Lambda + +TeX Live (including `latex` and `pdflatex`) for AWS Lambda, including packages and fonts required for Pandoc. + +Intended for instances powered by Amazon Linux 2.x, such as the `nodejs10.x` runtime, and the updated 2018.03 Amazon Linux 1 runtimes. + +The binaries will be in `/opt/texlive/bin/x86_64-linux` after linking the layer to a Lambda function. *Binaries depend on each other, so when executing, you will need to add that directory to your `PATH` environment variable*. + +``` +pdfTeX 3.14159265-2.6-1.40.20 (TeX Live 2019) +kpathsea version 6.3.1 +Compiled with libpng 1.6.36; using libpng 1.6.36 +Compiled with zlib 1.2.11; using zlib 1.2.11 +Compiled with xpdf version 4.01 +``` + +Note that this distribution comes with minimal fonts ([lm](https://ctan.org/pkg/lm?lang=en), [amsfonts](https://ctan.org/pkg/amsfonts?lang=en) and [ec](https://ctan.org/pkg/ec?lang=en)), required for Pandoc. To add more fonts, modify the build profile from . + +For an example of how to use the layer, check out +. diff --git a/README.md b/README.md new file mode 100644 index 0000000..923c553 --- /dev/null +++ b/README.md @@ -0,0 +1,95 @@ +# TeX Live (LaTeX/pdflatex) for AWS Lambda + +TeX Live (including `latex` and `pdflatex`) for AWS Lambda, including packages and fonts required for Pandoc. + +Intended for instances powered by Amazon Linux 2.x, such as the `nodejs10.x` runtime, and the updated 2018.03 Amazon Linux 1 runtimes + +## WORK IN PROGRESS!!! + +Note that this is work in progress, not ready for public usage yet. TeX Live requires a functioning `perl` executable, even at runtime, and this layer does not include that. + +## Usage + +After deployment, the binaries will be in `/opt/texlive/bin/x86_64-linux` after linking the layer to a Lambda function. + +Binaries depend on each other, so when executing, you will need to add that directory to your `PATH` environment variable. + +## Prerequisites + +* Docker desktop +* Unix Make environment +* AWS command line utilities (just for deployment) + +## Compiling the code + +* start Docker services +* `make all` + +There are two `make` scripts in this project. + +* [`Makefile`](Makefile) is intended to run on the build system, and just starts a Docker container matching the AWS Linux 2 environment for Lambda runtimes to compile Latex using the second script. +* [`Makefile_Latex`](Makefile_Latex) is the script that will run inside the container, and actually compile binaries. + +The output will be in the `result` dir. + +### Configuring the build + +By default, this compiles a version expecting to run as a Lambda layer from `/opt/texlive`. Change the expected runtime location in [`texlive.profile`](texlive.profile). + +The default Docker image used is `lambci/lambda-base-2:build`. To use a different base, provide a `DOCKER_IMAGE` variable when invoking `make`. + +You can include/exclude collections from [`texlive.profile`](texlive.profile), or change the CTAN packages installed in addition to the minimal collection in [`Makefile_Latex`](Makefile_Latex). + +Note that this distribution comes with minimal fonts ([lm](https://ctan.org/pkg/lm?lang=en), [amsfonts](https://ctan.org/pkg/amsfonts?lang=en) and [ec](https://ctan.org/pkg/ec?lang=en)) required for Pandoc. To add the full recommended font set, enable `collection-fontsrecommended` in [`texlive.profile`](texlive.profile). + +### Experimenting + +* `make bash` to open an interactive shell with all the build directories mounted + +### Compiled info + +``` +pdfTeX 3.14159265-2.6-1.40.20 (TeX Live 2019) +kpathsea version 6.3.1 +Compiled with libpng 1.6.36; using libpng 1.6.36 +Compiled with zlib 1.2.11; using zlib 1.2.11 +Compiled with xpdf version 4.01 +``` + +## Deploying to AWS as a layer + +Run the following command to deploy the compiled result as a layer in your AWS account. + +``` +make deploy DEPLOYMENT_BUCKET= +``` + +### configuring the deployment + +By default, this uses `latex-layer` as the stack name. Provide a `STACK_NAME` variable when +calling `make deploy` to use an alternative name. + +### example usage + +An example project is in the [example](example) directory. It sets up two buckets, and listens to file uploads on the first bucket to convert and generate HTML files from markdown. You can deploy it from the root Makefile using: + +``` +make deploy-example DEPLOYMENT_BUCKET= +``` + +For more information, check out: + +* general install process: +* profile variables: +* required packages for pandoc: + +## Author + +Gojko Adzic + +## License + +* These scripts: [MIT](https://opensource.org/licenses/MIT) +* TeX Live: +* LaTeX: +* Contained libraries all have separate licenses, check the respective web sites for more information diff --git a/example/.gitignore b/example/.gitignore new file mode 100644 index 0000000..c5bf9d0 --- /dev/null +++ b/example/.gitignore @@ -0,0 +1 @@ +output.yaml diff --git a/example/Makefile b/example/Makefile new file mode 100644 index 0000000..7b03222 --- /dev/null +++ b/example/Makefile @@ -0,0 +1,17 @@ +STACK_NAME ?= latex-layer-example +LATEX_STACK_NAME ?= latex-layer + +LATEX_LAYER ?=$(shell aws cloudformation describe-stacks --stack-name $(LATEX_STACK_NAME) --query Stacks[].Outputs[].OutputValue --output text) +SOURCES=$(shell find src/) + +clean: + rm -rf build + +output.yaml: template.yaml $(SOURCES) + mkdir -p build + aws cloudformation package --template-file $< --output-template-file $@ --s3-bucket $(DEPLOYMENT_BUCKET) + +deploy: output.yaml + aws cloudformation deploy --template-file $< --stack-name $(STACK_NAME) --capabilities CAPABILITY_IAM --parameter-overrides LatexLayer=$(LATEX_LAYER) + aws cloudformation describe-stacks --stack-name $(STACK_NAME) --query Stacks[].Outputs --output table + diff --git a/example/src/child-process-promise.js b/example/src/child-process-promise.js new file mode 100644 index 0000000..967b504 --- /dev/null +++ b/example/src/child-process-promise.js @@ -0,0 +1,26 @@ +/*global module, require, console, Promise */ +'use strict'; +const childProcess = require('child_process'), + spawnPromise = function (command, argsarray, envOptions) { + return new Promise((resolve, reject) => { + console.log('executing', command, argsarray.join(' ')); + const childProc = childProcess.spawn(command, argsarray, envOptions || {env: process.env, cwd: process.cwd()}), + resultBuffers = []; + childProc.stdout.on('data', buffer => { + console.log(buffer.toString()); + resultBuffers.push(buffer); + }); + childProc.stderr.on('data', buffer => console.error(buffer.toString())); + childProc.on('exit', (code, signal) => { + console.log(`${command} completed with ${code}:${signal}`); + if (code || signal) { + reject(`${command} failed with ${code || signal}`); + } else { + resolve(Buffer.concat(resultBuffers).toString().trim()); + } + }); + }); + }; +module.exports = { + spawn: spawnPromise +}; diff --git a/example/src/index.js b/example/src/index.js new file mode 100644 index 0000000..3771816 --- /dev/null +++ b/example/src/index.js @@ -0,0 +1,28 @@ + +const s3Util = require('./s3-util'), + childProcessPromise = require('./child-process-promise'), + path = require('path'), + os = require('os'), + OUTPUT_BUCKET = process.env.OUTPUT_BUCKET; + +exports.handler = function (eventObject, context) { + const eventRecord = eventObject.Records && eventObject.Records[0], + inputBucket = eventRecord.s3.bucket.name, + key = eventRecord.s3.object.key, + id = context.awsRequestId, + extension = '.pdf', + resultKey = key.replace(/\.[^.]+$/, extension), + workdir = os.tmpdir(), + inputFile = path.join(workdir, id + path.extname(key)), + outputFile = path.join(workdir, id + extension); + + + console.log('converting', inputBucket, key, 'using', inputFile); + return s3Util.downloadFileFromS3(inputBucket, key, inputFile) + .then(() => childProcessPromise.spawn( + 'pdflatex', + [inputFile], + {env: process.env, cwd: workdir} + )) + .then(() => s3Util.uploadFileToS3(OUTPUT_BUCKET, resultKey, outputFile, MIME_TYPE)); +}; diff --git a/example/src/s3-util.js b/example/src/s3-util.js new file mode 100644 index 0000000..7407a41 --- /dev/null +++ b/example/src/s3-util.js @@ -0,0 +1,38 @@ +/*global module, require, Promise, console */ + +const aws = require('aws-sdk'), + fs = require('fs'), + s3 = new aws.S3(), + downloadFileFromS3 = function (bucket, fileKey, filePath) { + 'use strict'; + console.log('downloading', bucket, fileKey, filePath); + return new Promise(function (resolve, reject) { + const file = fs.createWriteStream(filePath), + stream = s3.getObject({ + Bucket: bucket, + Key: fileKey + }).createReadStream(); + stream.on('error', reject); + file.on('error', reject); + file.on('finish', function () { + console.log('downloaded', bucket, fileKey); + resolve(filePath); + }); + stream.pipe(file); + }); + }, uploadFileToS3 = function (bucket, fileKey, filePath, contentType) { + 'use strict'; + console.log('uploading', bucket, fileKey, filePath); + return s3.upload({ + Bucket: bucket, + Key: fileKey, + Body: fs.createReadStream(filePath), + ACL: 'private', + ContentType: contentType + }).promise(); + }; + +module.exports = { + downloadFileFromS3: downloadFileFromS3, + uploadFileToS3: uploadFileToS3 +}; \ No newline at end of file diff --git a/example/template.yaml b/example/template.yaml new file mode 100644 index 0000000..d1c72b2 --- /dev/null +++ b/example/template.yaml @@ -0,0 +1,47 @@ +AWSTemplateFormatVersion: 2010-09-09 +Transform: AWS::Serverless-2016-10-31 +Description: > + Example project demonstrating the usage of the Latex Layer for AWS Linux 2 runtimes. + +Parameters: + LatexLayer: + Type: String +Resources: + UploadBucket: + Type: AWS::S3::Bucket + + ResultsBucket: + Type: AWS::S3::Bucket + + ConvertFileFunction: + Type: AWS::Serverless::Function + Properties: + Handler: index.handler + Timeout: 180 + MemorySize: 1024 + Runtime: nodejs10.x + CodeUri: src + Layers: + - !Ref LatexLayer + Policies: + - S3CrudPolicy: + BucketName: !Sub "${AWS::StackName}-*" + Environment: + Variables: + OUTPUT_BUCKET: !Ref ResultsBucket + PATH: '/opt/texlive/bin/x86_64-linux:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' + Events: + FileUpload: + Type: S3 + Properties: + Bucket: !Ref UploadBucket + Events: s3:ObjectCreated:* + +Outputs: + UploadBucket: + Description: "Upload S3 bucket" + Value: !Ref UploadBucket + ResultsBucket: + Description: "Results S3 bucket" + Value: !Ref ResultsBucket + diff --git a/template.yaml b/template.yaml new file mode 100644 index 0000000..4941acb --- /dev/null +++ b/template.yaml @@ -0,0 +1,40 @@ +AWSTemplateFormatVersion: 2010-09-09 +Transform: AWS::Serverless-2016-10-31 +Description: > + TeXLive (LaTeX, pdf-latex) for Amazon Linux 2, + including packages and fonts required for Pandoc. + + Check out https://github.com/serverlesspub/latex-aws-lambda-layer + for more information. +Resources: + LatexLayer: + Type: AWS::Serverless::LayerVersion + Properties: + LayerName: latex + Description: 'TeXLive (LaTeX, pdf-latex) for Amazon Linux 2' + ContentUri: build/layer.zip + CompatibleRuntimes: + - nodejs10.x + LicenseInfo: https://www.tug.org/texlive/LICENSE.TL + RetentionPolicy: Retain + +Outputs: + LayerVersion: + Description: Layer ARN Reference + Value: !Ref LatexLayer + +Metadata: + AWS::ServerlessRepo::Application: + Name: latex-lambda-layer + Description: > + TeXLive (LaTeX, pdf-latex) for Amazon Linux 2, + including packages and fonts required for Pandoc. + + Author: Gojko Adzic + SpdxLicenseId: Latex2e + LicenseUrl: LICENSE.md + ReadmeUrl: README-SAR.md + Labels: ['layer', 'latex', 'pdflatex', 'lambda', 'texlive', 'pandoc'] + HomePageUrl: https://github.com/serverlesspub/latex-aws-lambda-layer + SemanticVersion: 1.0.0 + SourceCodeUrl: https://github.com/serverlesspub/latex-aws-lambda-layer