Merge remote-tracking branch 'origin/master'

Fix #2: support pruning and forget via snapshot policies

README.md

@@ -1,7 +1,7 @@
 # autorestic
 High backup level CLI utility for [restic](https://restic.net/).
 
-Autorestic is a wrapper around the amazing [restic](https://restic.net/). While being amazing the restic cli can be a bit overwhelming and difficoult to manage if you habe many different location that you want to backup to multiple locations. This utility is aimed at making this easier 🙂
+Autorestic is a wrapper around the amazing [restic](https://restic.net/). While being amazing the restic cli can be a bit overwhelming and difficoult to manage if you have many different location that you want to backup to multiple locations. This utility is aimed at making this easier 🙂
 
 ![Sketch](./docs/Sketch.png)
 
@@ -10,13 +10,14 @@ Autorestic is a wrapper around the amazing [restic](https://restic.net/). While
 - Config files, no CLI
 - Predictable
 - Backup locations to multiple backends
+- Snapshot policies and pruning
 - Simple interface
 - Fully encrypted
 
 ## Installation
 
 ```
-curl -s https://raw.githubusercontent.com/CupCakeArmy/autorestic/master/install.sh | sh
+curl -s https://raw.githubusercontent.com/CupCakeArmy/autorestic/master/install.sh | bash
 ```
 
 ## 🚀 Quickstart
@@ -32,7 +33,7 @@ locations:
   home:
     from: /home/me
     to: remote
-  
+
   important:
     from: /path/to/important/stuff
     to:
@@ -45,7 +46,7 @@ backends:
     path: 'myBucket:backup/home'
     B2_ACCOUNT_ID: account_id
     B2_ACCOUNT_KEY: account_key
-  
+
   hdd:
     type: local
     path: /mnt/my_external_storage
@@ -53,11 +54,11 @@ backends:
 
 Then we check if everything is correct by running the `check` command. We will pass the `-a` (or `--all`) to tell autorestic to check all the locations.
 
-```
+If we would check only one location we could run the following: `autorestic check -l home`. Otherwise simpply check all locations with `autorestic check -a`
-autorestic check -a
-```
 
-If we would check only one location we could run the following: `autorestic -l home check`. 
+##### Note
+
+Note that the data is automatically encrypted on the server. The key will be generated and added to your config file. Every backend will have a separate key. You should keep a copy of the keys somewhere in case your server dies. Otherwise DATA IS LOST!
 
 ### Backup
 
@@ -68,9 +69,18 @@ autorestic backup -a
 ### Restore
 
 ```
-autorestic restore -a -- --target /path/where/to/restore
+autorestic restore -a --to /path/where/to/restore
 ```
 
+This will restore all the locations to the selected target. If for one location there are more than one backends specified autorestic will take the first one.
+
+Lets see a more realistic example (from the config above)
+```
+autorestic restore -l home --from hdd --to /path/where/to/restore
+```
+
+This will restore the location `home` to the `/path/where/to/restore` folder and taking the data from the backend `hdd`
+
 
 ## 🗂 Locations
 
@@ -108,7 +118,7 @@ For each backend you need to specify the right variables as shown in the example
 ##### `path`
 
 The path on the remote server.
-For object storages as 
+For object storages as
 
 ##### Example
 
@@ -121,9 +131,79 @@ backends:
     B2_ACCOUNT_KEY: backblaze_account_key
 ```
 
+### Pruning and snapshot policies
 
+Autorestic supports declaring snapshot policies for location to avoid keeping old snapshot around if you don't need them.
 
-##### Note
+This is based on [Restic's snapshots policies](https://restic.readthedocs.io/en/latest/060_forget.html#removing-snapshots-according-to-a-policy), and can be enabled for each location as shown below:
 
-Note that the data is automatically encrypted on the server. The key will be generated and added to your config file. Every backend will have a separate key. You should keep a copy of the keys somewhere in case your server dies. Otherwise DATA IS LOST!
+```yaml
+locations:
+  etc:
+    from: /etc
+    to: local
+    options:
+      forget:
+        keep-last: 5             # always keep at least 5 snapshots
+        keep-hourly: 3           # keep 3 last hourly shapshots
+        keep-daily: 4            # keep 4 last daily shapshots
+        keep-weekly: 1           # keep 1 last weekly shapshots
+        keep-monthly: 12         # keep 12 last monthly shapshots
+        keep-yearly: 7           # keep 7 last yearly shapshots
+        keep-within: "2w"        # keep snapshots from the last 2 weeks
+```
 
+Pruning can be triggered using `autorestic forget -a`, for all locations, or selectively with `autorestic forget -l <location>`. **please note that contrary to the restic CLI, `restic forget` will call `restic prune` internally.**
+
+Run with the `--dry-run` flag to only print information about the process without actually pruning the snapshots. This is especially useful for debugging or testing policies:
+```
+$ autorestic forget -a --dry-run --verbose
+
+Configuring Backends
+local : Done ✓
+
+Removing old shapshots according to policy
+etc ▶ local : Removing old spnapshots… ⏳
+etc ▶ local : Running in dry-run mode, not touching data
+etc ▶ local : Forgeting old snapshots… ⏳Applying Policy: all snapshots within 2d of the newest
+keep 3 snapshots:
+ID        Time                 Host        Tags        Reasons    Paths
+-----------------------------------------------------------------------------
+531b692a  2019-12-02 12:07:28  computer                within 2w  /etc
+51659674  2019-12-02 12:08:46  computer                within 2w  /etc
+f8f8f976  2019-12-02 12:11:08  computer                within 2w  /etc
+-----------------------------------------------------------------------------
+3 snapshots
+```
+
+### Excluding files/folders
+
+If you want to exclude certain files or folders it done easily by specifiyng the right flags in the location you desire to filter. The flags are taken straight from the [restic cli exclude rules](https://restic.readthedocs.io/en/latest/040_backup.html#excluding-files).
+
+```yaml
+locations:
+  my-location:
+    from: /data
+    to:
+      - local
+      - remote
+    options:
+      backup:
+        exclude:
+          - '*.nope'
+          - '*.abc'
+        exclude-file: .gitignore
+
+backends:
+  local:
+    ...
+   remote:
+    ...
+```
+
+## Contributors
+
+This amazing people helped the project!
+
+- @ChanceM [Docs]
+- @EliotBerriot [Docs, Pruning, S3]

install.sh

@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/bin/bash
 
 OUT_FILE=/usr/local/bin/autorestic
 

package.json

@@ -4,16 +4,15 @@
 		"build": "tsc",
 		"build:watch": "tsc -w",
 		"dev": "tsnd --no-notify --respawn ./src/autorestic.ts",
-		"bin": "npm run build && pkg lib/autorestic.js --targets latest-macos-x64,latest-linux-x64 --out-path bin"
+		"bin": "yarn run build && pkg lib/autorestic.js --targets latest-macos-x64,latest-linux-x64 --out-path bin"
 	},
 	"devDependencies": {
-		"@types/decompress": "^4.2.3",
 		"@types/js-yaml": "^3.12.1",
 		"@types/minimist": "^1.2.0",
 		"@types/node": "^12.11.7",
 		"pkg": "^4.4.0",
 		"ts-node-dev": "^1.0.0-pre.40",
-		"typescript": "^3.5.1"
+		"typescript": "^3.7"
 	},
 	"dependencies": {
 		"axios": "^0.19.0",

src/autorestic.ts

@@ -5,36 +5,41 @@ import { init } from './config'
 import handlers, { error, help } from './handlers'
 import { Config } from './types'
 
+
+
 process.on('uncaughtException', err => {
-  console.log(err.message)
+	console.log(err.message)
-  process.exit(1)
+	process.exit(1)
 })
 
 export const { _: commands, ...flags } = minimist(process.argv.slice(2), {
-  alias: {
+	alias: {
-    c: 'config',
+		c: 'config',
-    v: 'version',
+		v: 'version',
-    h: 'help',
+		h: 'help',
-    a: 'all',
+		a: 'all',
-    l: 'location',
+		l: 'location',
-    b: 'backend',
+		b: 'backend',
-  },
+		d: 'dry-run',
-  boolean: ['a'],
+	},
-  string: ['l', 'b'],
+	boolean: ['a', 'd'],
+	string: ['l', 'b'],
 })
 
-export const VERSION = '0.6'
+export const VERSION = '0.8'
 export const INSTALL_DIR = '/usr/local/bin'
 export const VERBOSE = flags.verbose
 
 export const config = init()
 
-function main() {
-  if (commands.length < 1) return help()
 
-  const command: string = commands[0]
+function main() {
-  const args: string[] = commands.slice(1)
+	if (commands.length < 1) return help()
-  ;(handlers[command] || error)(args, flags)
+
+	const command: string = commands[0]
+	const args: string[] = commands.slice(1)
+	;(handlers[command] || error)(args, flags)
 }
 
+
 main()

src/backend.ts

@@ -1,58 +1,67 @@
 import { Writer } from 'clitastic'
 
 import { config, VERBOSE } from './autorestic'
-import { Backend, Backends } from './types'
+import { Backend, Backends, Locations } from './types'
 import { exec, ConfigError } from './utils'
 
-const ALREADY_EXISTS = /(?=.*exists)(?=.*already)(?=.*config).*/
+
+
+const ALREADY_EXISTS = /(?=.*already)(?=.*config).*/
 
 export const getPathFromBackend = (backend: Backend): string => {
-  switch (backend.type) {
+	switch (backend.type) {
-    case 'local':
+		case 'local':
-      return backend.path
+			return backend.path
-    case 'b2':
+		case 'b2':
-    case 'azure':
+		case 'azure':
-    case 'gs':
+		case 'gs':
-    case 's3':
+		case 's3':
-      return `${backend.type}:${backend.path}`
+			return `${backend.type}:${backend.path}`
-    case 'sftp':
+		case 'sftp':
-    case 'rest':
+		case 'rest':
-      throw new Error(`Unsupported backend type: "${backend.type}"`)
+			throw new Error(`Unsupported backend type: "${backend.type}"`)
-    default:
+		default:
-      throw new Error(`Unknown backend type.`)
+			throw new Error(`Unknown backend type.`)
-  }
+	}
 }
 
 export const getEnvFromBackend = (backend: Backend) => {
-  const { type, path, key, ...rest } = backend
+	const { type, path, key, ...rest } = backend
-  return {
+	return {
-    RESTIC_PASSWORD: key,
+		RESTIC_PASSWORD: key,
-    RESTIC_REPOSITORY: getPathFromBackend(backend),
+		RESTIC_REPOSITORY: getPathFromBackend(backend),
-    ...rest,
+		...rest,
-  }
+	}
+}
+
+export const getBackendsFromLocations = (locations: Locations): string[] => {
+	const backends = new Set<string>()
+	for (const to of Object.values(locations).map(location => location.to))
+		Array.isArray(to) ? to.forEach(t => backends.add(t)) : backends.add(to)
+	return Array.from(backends)
 }
 
 export const checkAndConfigureBackend = (name: string, backend: Backend) => {
-  const writer = new Writer(name.blue + ' : ' + 'Configuring... ⏳')
+	const writer = new Writer(name.blue + ' : ' + 'Configuring... ⏳')
-  const env = getEnvFromBackend(backend)
+	const env = getEnvFromBackend(backend)
 
-  const { out, err } = exec('restic', ['init'], { env })
+	const { out, err } = exec('restic', ['init'], { env })
 
-  if (err.length > 0 && !ALREADY_EXISTS.test(err))
+	if (err.length > 0 && !ALREADY_EXISTS.test(err))
-    throw new Error(`Could not load the backend "${name}": ${err}`)
+		throw new Error(`Could not load the backend "${name}": ${err}`)
 
-  if (VERBOSE && out.length > 0) console.log(out)
+	if (VERBOSE && out.length > 0) console.log(out)
 
-  writer.done(name.blue + ' : ' + 'Done ✓'.green)
+	writer.done(name.blue + ' : ' + 'Done ✓'.green)
 }
 
 export const checkAndConfigureBackends = (backends?: Backends) => {
-  if (!backends) {
+	if (!backends) {
-    if (!config) throw ConfigError
+		if (!config) throw ConfigError
-    backends = config.backends
+		backends = config.backends
-  }
+	}
 
-  console.log('\nConfiguring Backends'.grey.underline)
+	console.log('\nConfiguring Backends'.grey.underline)
-  for (const [name, backend] of Object.entries(backends))
+	for (const [name, backend] of Object.entries(backends))
-    checkAndConfigureBackend(name, backend)
+		checkAndConfigureBackend(name, backend)
 }

src/backup.ts

@@ -3,49 +3,45 @@ import { Writer } from 'clitastic'
 import { config, VERBOSE } from './autorestic'
 import { getEnvFromBackend } from './backend'
 import { Locations, Location } from './types'
-import { exec, ConfigError } from './utils'
+import { exec, ConfigError, pathRelativeToConfigFile, getFlagsFromLocation } from './utils'
-import { CONFIG_FILE } from './config'
-import { resolve, dirname, isAbsolute } from 'path'
 
-export const backupSingle = (name: string, from: string, to: string) => {
-  if (!config) throw ConfigError
-  const writer = new Writer(name + to.blue + ' : ' + 'Backing up... ⏳')
-  const backend = config.backends[to]
 
-  // Check if is an absolute path, otherwise get the path relative to the config file
-  const pathRelativeToConfigFile = isAbsolute(from)
-    ? from
-    : resolve(dirname(CONFIG_FILE), from)
 
-  const cmd = exec('restic', ['backup', pathRelativeToConfigFile], {
+export const backupSingle = (name: string, to: string, location: Location) => {
-    env: getEnvFromBackend(backend),
+	if (!config) throw ConfigError
-  })
+	const writer = new Writer(name + to.blue + ' : ' + 'Backing up... ⏳')
 
-  if (VERBOSE) console.log(cmd.out, cmd.err)
+	const backend = config.backends[to]
-  writer.done(name + to.blue + ' : ' + 'Done ✓'.green)
+	const path = pathRelativeToConfigFile(location.from)
+
+	const cmd = exec(
+		'restic',
+		['backup', path, ...getFlagsFromLocation(location, 'backup')],
+		{ env: getEnvFromBackend(backend) },
+	)
+
+	if (VERBOSE) console.log(cmd.out, cmd.err)
+	writer.done(name + to.blue + ' : ' + 'Done ✓'.green)
 }
 
-export const backupLocation = (name: string, backup: Location) => {
+export const backupLocation = (name: string, location: Location) => {
-  const display = name.yellow + ' ▶ '
+	const display = name.yellow + ' ▶ '
-  if (Array.isArray(backup.to)) {
+	const filler = new Array(name.length + 3).fill(' ').join('')
-    let first = true
+	let first = true
-    for (const t of backup.to) {
+
-      const nameOrBlankSpaces: string = first
+	for (const t of Array.isArray(location.to) ? location.to : [location.to]) {
-        ? display
+		backupSingle(first ? display : filler, t, location)
-        : new Array(name.length + 3).fill(' ').join('')
+		if (first) first = false
-      backupSingle(nameOrBlankSpaces, backup.from, t)
+	}
-      if (first) first = false
-    }
-  } else backupSingle(display, backup.from, backup.to)
 }
 
-export const backupAll = (backups?: Locations) => {
+export const backupAll = (locations?: Locations) => {
-  if (!backups) {
+	if (!locations) {
-    if (!config) throw ConfigError
+		if (!config) throw ConfigError
-    backups = config.locations
+		locations = config.locations
-  }
+	}
 
-  console.log('\nBacking Up'.underline.grey)
+	console.log('\nBacking Up'.underline.grey)
-  for (const [name, backup] of Object.entries(backups))
+	for (const [name, location] of Object.entries(locations))
-    backupLocation(name, backup)
+		backupLocation(name, location)
 }

src/config.ts

@@ -6,81 +6,84 @@ import { Backend, Config } from './types'
 import { makeObjectKeysLowercase, rand } from './utils'
 import { homedir } from 'os'
 
+
+
 export const normalizeAndCheckBackends = (config: Config) => {
-  config.backends = makeObjectKeysLowercase(config.backends)
+	config.backends = makeObjectKeysLowercase(config.backends)
 
-  for (const [name, { type, path, key, ...rest }] of Object.entries(
+	for (const [name, { type, path, key, ...rest }] of Object.entries(
-    config.backends
+		config.backends,
-  )) {
+	)) {
-    if (!type || !path)
+		if (!type || !path)
-      throw new Error(
+			throw new Error(
-        `The backend "${name}" is missing some required attributes`
+				`The backend "${name}" is missing some required attributes`,
-      )
+			)
 
-    const tmp: any = {
+		const tmp: any = {
-      type,
+			type,
-      path,
+			path,
-      key: key || rand(128),
+			key: key || rand(128),
-    }
+		}
-    for (const [key, value] of Object.entries(rest))
+		for (const [key, value] of Object.entries(rest))
-      tmp[key.toUpperCase()] = value
+			tmp[key.toUpperCase()] = value
 
-    config.backends[name] = tmp as Backend
+		config.backends[name] = tmp as Backend
-  }
+	}
 }
 
 export const normalizeAndCheckBackups = (config: Config) => {
-  config.locations = makeObjectKeysLowercase(config.locations)
+	config.locations = makeObjectKeysLowercase(config.locations)
-  const backends = Object.keys(config.backends)
+	const backends = Object.keys(config.backends)
 
-  const checkDestination = (backend: string, backup: string) => {
+	const checkDestination = (backend: string, backup: string) => {
-    if (!backends.includes(backend))
+		if (!backends.includes(backend))
-      throw new Error(`Cannot find the backend "${backend}" for "${backup}"`)
+			throw new Error(`Cannot find the backend "${backend}" for "${backup}"`)
-  }
+	}
 
-  for (const [name, { from, to, ...rest }] of Object.entries(
+	for (const [name, { from, to, ...rest }] of Object.entries(
-    config.locations
+		config.locations,
-  )) {
+	)) {
-    if (!from || !to)
+		if (!from || !to)
-      throw new Error(
+			throw new Error(
-        `The backup "${name}" is missing some required attributes`
+				`The backup "${name}" is missing some required attributes`,
-      )
+			)
 
-    if (Array.isArray(to)) for (const t of to) checkDestination(t, name)
+		if (Array.isArray(to)) for (const t of to) checkDestination(t, name)
-    else checkDestination(to, name)
+		else checkDestination(to, name)
-  }
+	}
 }
 
 const findConfigFile = (): string | undefined => {
-  const config = '.autorestic.yml'
+	const config = '.autorestic.yml'
-  const paths = [
+	const paths = [
-    resolve(flags.config || ''),
+		resolve(flags.config || ''),
-    resolve('./' + config),
+		resolve('./' + config),
-    homedir() + '/' + config,
+		homedir() + '/' + config,
-  ]
+	]
-  for (const path of paths) {
+	for (const path of paths) {
-    try {
+		try {
-      const file = statSync(path)
+			const file = statSync(path)
-      if (file.isFile()) return path
+			if (file.isFile()) return path
-    } catch (e) {}
+		} catch (e) {
-  }
+		}
+	}
 }
 
 export let CONFIG_FILE: string = ''
 
 export const init = (): Config | undefined => {
-  const file = findConfigFile()
+	const file = findConfigFile()
-  if (file) CONFIG_FILE = file
+	if (file) CONFIG_FILE = file
-  else return
+	else return
 
-  const raw: Config = makeObjectKeysLowercase(
+	const raw: Config = makeObjectKeysLowercase(
-    yaml.safeLoad(readFileSync(CONFIG_FILE).toString())
+		yaml.safeLoad(readFileSync(CONFIG_FILE).toString()),
-  )
+	)
 
-  normalizeAndCheckBackends(raw)
+	normalizeAndCheckBackends(raw)
-  normalizeAndCheckBackups(raw)
+	normalizeAndCheckBackups(raw)
 
-  writeFileSync(CONFIG_FILE, yaml.safeDump(raw))
+	writeFileSync(CONFIG_FILE, yaml.safeDump(raw))
 
-  return raw
+	return raw
 }

src/forget.ts

@@ -0,0 +1,60 @@
+import { Writer } from 'clitastic'
+
+import { config, VERBOSE } from './autorestic'
+import { getEnvFromBackend } from './backend'
+import { Locations, Location, Flags } from './types'
+import { exec, ConfigError, pathRelativeToConfigFile, getFlagsFromLocation } from './utils'
+
+
+
+export const forgetSingle = (name: string, to: string, location: Location, dryRun: boolean) => {
+	if (!config) throw ConfigError
+	const base = name + to.blue + ' : '
+	const writer = new Writer(base + 'Removing old snapshots… ⏳')
+
+	const backend = config.backends[to]
+	const path = pathRelativeToConfigFile(location.from)
+	const flags = getFlagsFromLocation(location, 'forget')
+
+	if (flags.length == 0) {
+		writer.done(base + 'skipping, no policy declared')
+		return
+	}
+	if (dryRun) flags.push('--dry-run')
+
+	writer.replaceLn(base + 'Forgetting old snapshots… ⏳')
+	const cmd = exec(
+		'restic',
+		['forget', '--path', path, '--prune', ...flags],
+		{ env: getEnvFromBackend(backend) },
+	)
+
+	if (VERBOSE) console.log(cmd.out, cmd.err)
+	writer.done(base + 'Done ✓'.green)
+}
+
+export const forgetLocation = (name: string, backup: Location, dryRun: boolean) => {
+	const display = name.yellow + ' ▶ '
+	const filler = new Array(name.length + 3).fill(' ').join('')
+	let first = true
+
+	for (const t of Array.isArray(backup.to) ? backup.to : [backup.to]) {
+		const nameOrBlankSpaces: string = first ? display : filler
+		forgetSingle(nameOrBlankSpaces, t, backup, dryRun)
+		if (first) first = false
+	}
+}
+
+export const forgetAll = (backups?: Locations, flags?: Flags) => {
+	if (!config) throw ConfigError
+	if (!backups) {
+		backups = config.locations
+	}
+
+	console.log('\nRemoving old snapshots according to policy'.underline.grey)
+	const dryRun = flags ? flags['dry-run'] : false
+	if (dryRun) console.log('Running in dry-run mode, not touching data\n'.yellow)
+
+	for (const [name, backup] of Object.entries(backups))
+		forgetLocation(name, backup, dryRun)
+}

src/handlers.ts

@@ -5,240 +5,266 @@ import { tmpdir } from 'os'
 import { join, resolve } from 'path'
 
 import { config, INSTALL_DIR, VERSION } from './autorestic'
-import { checkAndConfigureBackends, getEnvFromBackend } from './backend'
+import { checkAndConfigureBackends, getBackendsFromLocations, getEnvFromBackend } from './backend'
 import { backupAll } from './backup'
-import { Backends, Flags, Locations } from './types'
+import { forgetAll } from './forget'
+import { Backend, Backends, Flags, Locations } from './types'
 import {
-  checkIfCommandIsAvailable,
+	checkIfCommandIsAvailable,
-  checkIfResticIsAvailable,
+	checkIfResticIsAvailable,
-  downloadFile,
+	downloadFile,
-  exec,
+	exec,
-  filterObjectByKey,
+	filterObjectByKey,
-  singleToArray,
+	singleToArray,
-  ConfigError,
+	ConfigError,
 } from './utils'
 
+
+
 export type Handlers = {
-  [command: string]: (args: string[], flags: Flags) => void
+	[command: string]: (args: string[], flags: Flags) => void
 }
 
 const parseBackend = (flags: Flags): Backends => {
-  if (!config) throw ConfigError
+	if (!config) throw ConfigError
-  if (!flags.all && !flags.backend)
+	if (!flags.all && !flags.backend)
-    throw new Error(
+		throw new Error(
-      'No backends specified.'.red +
+			'No backends specified.'.red +
-        '\n--all [-a]\t\t\t\tCheck all.' +
+			'\n--all [-a]\t\t\t\tCheck all.' +
-        '\n--backend [-b] myBackend\t\tSpecify one or more backend'
+			'\n--backend [-b] myBackend\t\tSpecify one or more backend',
-    )
+		)
-  if (flags.all) return config.backends
+	if (flags.all) return config.backends
-  else {
+	else {
-    const backends = singleToArray<string>(flags.backend)
+		const backends = singleToArray<string>(flags.backend)
-    for (const backend of backends)
+		for (const backend of backends)
-      if (!config.backends[backend])
+			if (!config.backends[backend])
-        throw new Error('Invalid backend: '.red + backend)
+				throw new Error('Invalid backend: '.red + backend)
-    return filterObjectByKey(config.backends, backends)
+		return filterObjectByKey(config.backends, backends)
-  }
+	}
 }
 
 const parseLocations = (flags: Flags): Locations => {
-  if (!config) throw ConfigError
+	if (!config) throw ConfigError
-  if (!flags.all && !flags.location)
+	if (!flags.all && !flags.location)
-    throw new Error(
+		throw new Error(
-      'No locations specified.'.red +
+			'No locations specified.'.red +
-        '\n--all [-a]\t\t\t\tBackup all.' +
+			'\n--all [-a]\t\t\t\tBackup all.' +
-        '\n--location [-l] site1\t\t\tSpecify one or more locations'
+			'\n--location [-l] site1\t\t\tSpecify one or more locations',
-    )
+		)
 
-  if (flags.all) {
+	if (flags.all) {
-    return config.locations
+		return config.locations
-  } else {
+	} else {
-    const locations = singleToArray<string>(flags.location)
+		const locations = singleToArray<string>(flags.location)
-    for (const location of locations)
+		for (const location of locations)
-      if (!config.locations[location])
+			if (!config.locations[location])
-        throw new Error('Invalid location: '.red + location)
+				throw new Error('Invalid location: '.red + location)
-    return filterObjectByKey(config.locations, locations)
+		return filterObjectByKey(config.locations, locations)
-  }
+	}
 }
 
 const handlers: Handlers = {
-  check(args, flags) {
+	check(args, flags) {
-    checkIfResticIsAvailable()
+		checkIfResticIsAvailable()
-    const backends = parseBackend(flags)
+		const backends = parseBackend(flags)
-    checkAndConfigureBackends(backends)
+		checkAndConfigureBackends(backends)
-  },
+	},
-  backup(args, flags) {
+	backup(args, flags) {
-    if (!config) throw ConfigError
+		if (!config) throw ConfigError
-    checkIfResticIsAvailable()
+		checkIfResticIsAvailable()
-    const locations: Locations = parseLocations(flags)
+		const locations: Locations = parseLocations(flags)
 
-    const backends = new Set<string>()
+		checkAndConfigureBackends(
-    for (const to of Object.values(locations).map(location => location.to))
+			filterObjectByKey(config.backends, getBackendsFromLocations(locations)),
-      Array.isArray(to) ? to.forEach(t => backends.add(t)) : backends.add(to)
+		)
+		backupAll(locations)
 
-    checkAndConfigureBackends(
+		console.log('\nFinished!'.underline + ' 🎉')
-      filterObjectByKey(config.backends, Array.from(backends))
+	},
-    )
+	restore(args, flags) {
-    backupAll(locations)
+		if (!config) throw ConfigError
+		checkIfResticIsAvailable()
 
-    console.log('\nFinished!'.underline + ' 🎉')
+		if (!flags.to) {
-  },
+			console.log(`You need to specify the restore path with --to`.red)
-  restore(args, flags) {
+			return
-    if (!config) throw ConfigError
+		}
-    checkIfResticIsAvailable()
-    const locations = parseLocations(flags)
-    for (const [name, location] of Object.entries(locations)) {
-      const w = new Writer(name.green + `\t\tRestoring... ⏳`)
-      const env = getEnvFromBackend(
-        config.backends[
-          Array.isArray(location.to) ? location.to[0] : location.to
-        ]
-      )
 
-      exec(
+		const locations = parseLocations(flags)
-        'restic',
+		for (const [name, location] of Object.entries(locations)) {
-        ['restore', 'latest', '--path', resolve(location.from), ...args],
+			const baseText = name.green + '\t\t'
-        { env }
+			const w = new Writer(baseText + `Starting...`)
-      )
-      w.done(name.green + '\t\tDone 🎉')
-    }
-  },
-  exec(args, flags) {
-    checkIfResticIsAvailable()
-    const backends = parseBackend(flags)
-    for (const [name, backend] of Object.entries(backends)) {
-      console.log(`\n${name}:\n`.grey.underline)
-      const env = getEnvFromBackend(backend)
 
-      const { out, err } = exec('restic', args, { env })
+			let backend: string = Array.isArray(location.to) ? location.to[0] : location.to
-      console.log(out, err)
+			if (flags.from) {
-    }
+				if (!location.to.includes(flags.from)) {
-  },
+					w.done(baseText + `Backend ${flags.from} is not a valid location for ${name}`.red)
-  async install() {
+					continue
-    try {
+				}
-      checkIfResticIsAvailable()
+				backend = flags.from
-      console.log('Restic is already installed')
+				w.replaceLn(baseText + `Restoring from ${backend.blue}...`)
-      return
+			} else if (Array.isArray(location.to) && location.to.length > 1) {
-    } catch (e) {}
+				w.replaceLn(baseText + `Restoring from ${backend.blue}...\tTo select a specific backend pass the ${'--from'.blue} flag`)
+			}
+			const env = getEnvFromBackend(config.backends[backend])
 
-    const w = new Writer('Checking latest version... ⏳')
+			exec(
-    checkIfCommandIsAvailable('bzip2')
+				'restic',
-    const { data: json } = await axios({
+				['restore', 'latest', '--path', resolve(location.from), '--target', flags.to],
-      method: 'get',
+				{ env },
-      url: 'https://api.github.com/repos/restic/restic/releases/latest',
+			)
-      responseType: 'json',
+			w.done(name.green + '\t\tDone 🎉')
-    })
+		}
+	},
+	forget(args, flags) {
+		if (!config) throw ConfigError
+		checkIfResticIsAvailable()
+		const locations: Locations = parseLocations(flags)
 
-    const archMap: { [a: string]: string } = {
+		checkAndConfigureBackends(
-      x32: '386',
+			filterObjectByKey(config.backends, getBackendsFromLocations(locations)),
-      x64: 'amd64',
+		)
-    }
+		forgetAll(locations, flags)
 
-    w.replaceLn('Downloading binary... 🌎')
+		console.log('\nFinished!'.underline + ' 🎉')
-    const name = `${json.name.replace(' ', '_')}_${process.platform}_${
+	},
-      archMap[process.arch]
+	exec(args, flags) {
-    }.bz2`
+		checkIfResticIsAvailable()
-    const dl = json.assets.find((asset: any) => asset.name === name)
+		const backends = parseBackend(flags)
-    if (!dl)
+		for (const [name, backend] of Object.entries(backends)) {
-      return console.log(
+			console.log(`\n${name}:\n`.grey.underline)
-        'Cannot get the right binary.'.red,
+			const env = getEnvFromBackend(backend)
-        'Please see https://bit.ly/2Y1Rzai'
-      )
 
-    const tmp = join(tmpdir(), name)
+			const { out, err } = exec('restic', args, { env })
-    const extracted = tmp.slice(0, -4) //without the .bz2
+			console.log(out, err)
+		}
+	},
+	async install() {
+		try {
+			checkIfResticIsAvailable()
+			console.log('Restic is already installed')
+			return
+		} catch (e) {
+		}
 
-    await downloadFile(dl.browser_download_url, tmp)
+		const w = new Writer('Checking latest version... ⏳')
+		checkIfCommandIsAvailable('bzip2')
+		const { data: json } = await axios({
+			method: 'get',
+			url: 'https://api.github.com/repos/restic/restic/releases/latest',
+			responseType: 'json',
+		})
 
-    // TODO: Native bz2
+		const archMap: { [a: string]: string } = {
-    // Decompress
+			x32: '386',
-    w.replaceLn('Decompressing binary... 📦')
+			x64: 'amd64',
-    exec('bzip2', ['-dk', tmp])
+		}
-    unlinkSync(tmp)
 
-    w.replaceLn(`Moving to ${INSTALL_DIR} 🚙`)
+		w.replaceLn('Downloading binary... 🌎')
-    exec('chmod', ['+x', extracted])
+		const name = `${json.name.replace(' ', '_')}_${process.platform}_${
-    exec('mv', [extracted, INSTALL_DIR + '/restic'])
+			archMap[process.arch]
+		}.bz2`
+		const dl = json.assets.find((asset: any) => asset.name === name)
+		if (!dl)
+			return console.log(
+				'Cannot get the right binary.'.red,
+				'Please see https://bit.ly/2Y1Rzai',
+			)
 
-    w.done(
+		const tmp = join(tmpdir(), name)
-      `\nFinished! restic is installed under: ${INSTALL_DIR}`.underline + ' 🎉'
+		const extracted = tmp.slice(0, -4) //without the .bz2
-    )
-  },
-  uninstall() {
-    for (const bin of ['restic', 'autorestic'])
-      try {
-        unlinkSync(INSTALL_DIR + '/' + bin)
-        console.log(`Finished! ${bin} was uninstalled`)
-      } catch (e) {
-        console.log(`${bin} is already uninstalled`.red)
-      }
-  },
-  async update() {
-    checkIfResticIsAvailable()
-    const w = new Writer('Checking for latest restic version... ⏳')
-    exec('restic', ['self-update'])
 
-    w.replaceLn('Checking for latest autorestic version... ⏳')
+		await downloadFile(dl.browser_download_url, tmp)
-    const { data: json } = await axios({
-      method: 'get',
-      url:
-        'https://api.github.com/repos/cupcakearmy/autorestic/releases/latest',
-      responseType: 'json',
-    })
 
-    if (json.tag_name != VERSION) {
+		w.replaceLn('Decompressing binary... 📦')
-      const platformMap: { [key: string]: string } = {
+		exec('bzip2', ['-dk', tmp])
-        darwin: 'macos',
+		unlinkSync(tmp)
-      }
 
-      const name = `autorestic_${platformMap[process.platform] ||
+		w.replaceLn(`Moving to ${INSTALL_DIR} 🚙`)
-        process.platform}_${process.arch}`
+		exec('chmod', ['+x', extracted])
-      const dl = json.assets.find((asset: any) => asset.name === name)
+		exec('mv', [extracted, INSTALL_DIR + '/restic'])
 
-      const to = INSTALL_DIR + '/autorestic'
+		w.done(
-      w.replaceLn('Downloading binary... 🌎')
+			`\nFinished! restic is installed under: ${INSTALL_DIR}`.underline + ' 🎉',
-      await downloadFile(dl.browser_download_url, to)
+		)
+	},
+	uninstall() {
+		for (const bin of ['restic', 'autorestic'])
+			try {
+				unlinkSync(INSTALL_DIR + '/' + bin)
+				console.log(`Finished! ${bin} was uninstalled`)
+			} catch (e) {
+				console.log(`${bin} is already uninstalled`.red)
+			}
+	},
+	async update() {
+		checkIfResticIsAvailable()
+		const w = new Writer('Checking for latest restic version... ⏳')
+		exec('restic', ['self-update'])
 
-      exec('chmod', ['+x', to])
+		w.replaceLn('Checking for latest autorestic version... ⏳')
-    }
+		const { data: json } = await axios({
+			method: 'get',
+			url:
+				'https://api.github.com/repos/cupcakearmy/autorestic/releases/latest',
+			responseType: 'json',
+		})
 
-    w.done('All up to date! 🚀')
+		if (json.tag_name != VERSION) {
-  },
+			const platformMap: { [key: string]: string } = {
-  version() {
+				darwin: 'macos',
-    console.log('version'.grey, VERSION)
+			}
-  },
+
+			const name = `autorestic_${platformMap[process.platform] || process.platform}_${process.arch}`
+			const dl = json.assets.find((asset: any) => asset.name === name)
+
+			const to = INSTALL_DIR + '/autorestic'
+			w.replaceLn('Downloading binary... 🌎')
+			await downloadFile(dl.browser_download_url, to)
+
+			exec('chmod', ['+x', to])
+		}
+
+		w.done('All up to date! 🚀')
+	},
+	version() {
+		console.log('version'.grey, VERSION)
+	},
 }
 
 export const help = () => {
-  console.log(
+	console.log(
-    '\nAutorestic'.blue +
+		'\nAutorestic'.blue +
-      ` - ${VERSION} - Easy Restic CLI Utility` +
+		` - ${VERSION} - Easy Restic CLI Utility` +
-      '\n' +
+		'\n' +
-      '\nOptions:'.yellow +
+		'\nOptions:'.yellow +
-      `\n  -c, --config                                                          Specify config file. Default: .autorestic.yml` +
+		`\n  -c, --config                                                          Specify config file. Default: .autorestic.yml` +
-      '\n' +
+		'\n' +
-      '\nCommands:'.yellow +
+		'\nCommands:'.yellow +
-      '\n  check    [-b, --backend]  [-a, --all]                                 Check backends' +
+		'\n  check    [-b, --backend]  [-a, --all]                                 Check backends' +
-      '\n  backup   [-l, --location] [-a, --all]                                 Backup all or specified locations' +
+		'\n  backup   [-l, --location] [-a, --all]                                 Backup all or specified locations' +
-      '\n  restore  [-l, --location] [-- --target <out dir>]                     Restore all or specified locations' +
+		'\n  forget   [-l, --location] [-a, --all] [--dry-run]                     Forget old snapshots according to declared policies' +
-      '\n' +
+		'\n  restore  [-l, --location] [--from backend] [--to <out dir>]           Restore all or specified locations' +
-      '\n  exec     [-b, --backend]  [-a, --all] <command> -- [native options]   Execute native restic command' +
+		'\n' +
-      '\n' +
+		'\n  exec     [-b, --backend]  [-a, --all] <command> -- [native options]   Execute native restic command' +
-      '\n  install                                                               install restic' +
+		'\n' +
-      '\n  uninstall                                                             uninstall restic' +
+		'\n  install                                                               install restic' +
-      '\n  update                                                                update restic' +
+		'\n  uninstall                                                             uninstall restic' +
-      '\n  help                                                                  Show help' +
+		'\n  update                                                                update restic' +
-      '\n' +
+		'\n  help                                                                  Show help' +
-      '\nExamples: '.yellow +
+		'\n' +
-      'https://git.io/fjVbg' +
+		'\nExamples: '.yellow +
-      '\n'
+		'https://git.io/fjVbg' +
-  )
+		'\n',
+	)
 }
+
 export const error = () => {
-  help()
+	help()
-  console.log(
+	console.log(
-    `Invalid Command:`.red.underline,
+		`Invalid Command:`.red.underline,
-    `${process.argv.slice(2).join(' ')}`
+		`${process.argv.slice(2).join(' ')}`,
-  )
+	)
 }
 
 export default handlers

src/types.ts

@@ -1,77 +1,94 @@
 type BackendLocal = {
-  type: 'local'
+	type: 'local'
-  key: string
+	key: string
-  path: string
+	path: string
 }
 
 type BackendSFTP = {
-  type: 'sftp'
+	type: 'sftp'
-  key: string
+	key: string
-  path: string
+	path: string
-  password?: string
+	password?: string
 }
 
 type BackendREST = {
-  type: 'rest'
+	type: 'rest'
-  key: string
+	key: string
-  path: string
+	path: string
-  user?: string
+	user?: string
-  password?: string
+	password?: string
 }
 
 type BackendS3 = {
-  type: 's3'
+	type: 's3'
-  key: string
+	key: string
-  path: string
+	path: string
-  aws_access_key_id: string
+	aws_access_key_id: string
-  aws_secret_access_key: string
+	aws_secret_access_key: string
 }
 
 type BackendB2 = {
-  type: 'b2'
+	type: 'b2'
-  key: string
+	key: string
-  path: string
+	path: string
-  b2_account_id: string
+	b2_account_id: string
-  b2_account_key: string
+	b2_account_key: string
 }
 
 type BackendAzure = {
-  type: 'azure'
+	type: 'azure'
-  key: string
+	key: string
-  path: string
+	path: string
-  azure_account_name: string
+	azure_account_name: string
-  azure_account_key: string
+	azure_account_key: string
 }
 
 type BackendGS = {
-  type: 'gs'
+	type: 'gs'
-  key: string
+	key: string
-  path: string
+	path: string
-  google_project_id: string
+	google_project_id: string
-  google_application_credentials: string
+	google_application_credentials: string
 }
 
 export type Backend =
-  | BackendAzure
+	| BackendAzure
-  | BackendB2
+	| BackendB2
-  | BackendGS
+	| BackendGS
-  | BackendLocal
+	| BackendLocal
-  | BackendREST
+	| BackendREST
-  | BackendS3
+	| BackendS3
-  | BackendSFTP
+	| BackendSFTP
 
 export type Backends = { [name: string]: Backend }
 
+export type ForgetPolicy = {
+	last?: number,
+	hourly?: number,
+	daily?: number,
+	weekly?: number,
+	monthly?: number,
+	yearly?: number,
+	within?: string,
+	tags?: string[],
+}
+
 export type Location = {
-  from: string
+	from: string
-  to: string | string[]
+	to: string | string[]
+	keep?: ForgetPolicy
+	options?: {
+		[key: string]: {
+			[key: string]: string | string[]
+		}
+	}
 }
 
 export type Locations = { [name: string]: Location }
 
 export type Config = {
-  locations: Locations
+	locations: Locations
-  backends: Backends
+	backends: Backends
 }
 
 export type Flags = { [arg: string]: any }

src/utils.ts

@@ -2,76 +2,108 @@ import axios from 'axios'
 import { spawnSync, SpawnSyncOptions } from 'child_process'
 import { randomBytes } from 'crypto'
 import { createWriteStream } from 'fs'
+import { isAbsolute, resolve, dirname } from 'path'
+import { CONFIG_FILE } from './config'
+import { Location } from './types'
+
+
 
 export const exec = (
-  command: string,
+	command: string,
-  args: string[],
+	args: string[],
-  { env, ...rest }: SpawnSyncOptions = {}
+	{ env, ...rest }: SpawnSyncOptions = {},
 ) => {
-  const cmd = spawnSync(command, args, {
+	const cmd = spawnSync(command, args, {
-    ...rest,
+		...rest,
-    env: {
+		env: {
-      ...process.env,
+			...process.env,
-      ...env,
+			...env,
-    },
+		},
-  })
+	})
 
-  const out = cmd.stdout && cmd.stdout.toString().trim()
+	const out = cmd.stdout && cmd.stdout.toString().trim()
-  const err = cmd.stderr && cmd.stderr.toString().trim()
+	const err = cmd.stderr && cmd.stderr.toString().trim()
 
-  return { out, err }
+	return { out, err }
 }
 
 export const checkIfResticIsAvailable = () =>
-  checkIfCommandIsAvailable(
+	checkIfCommandIsAvailable(
-    'restic',
+		'restic',
-    'Restic is not installed'.red +
+		'Restic is not installed'.red +
-      ' https://restic.readthedocs.io/en/latest/020_installation.html#stable-releases'
+		' https://restic.readthedocs.io/en/latest/020_installation.html#stable-releases',
-  )
+	)
 
 export const checkIfCommandIsAvailable = (cmd: string, errorMsg?: string) => {
-  if (require('child_process').spawnSync(cmd).error)
+	if (require('child_process').spawnSync(cmd).error)
-    throw new Error(errorMsg ? errorMsg : `"${errorMsg}" is not installed`.red)
+		throw new Error(errorMsg ? errorMsg : `"${errorMsg}" is not installed`.red)
 }
 
 export const makeObjectKeysLowercase = (object: Object): any =>
-  Object.fromEntries(
+	Object.fromEntries(
-    Object.entries(object).map(([key, value]) => [key.toLowerCase(), value])
+		Object.entries(object).map(([key, value]) => [key.toLowerCase(), value]),
-  )
+	)
+
 
 export function rand(length = 32): string {
-  return randomBytes(length / 2).toString('hex')
+	return randomBytes(length / 2).toString('hex')
 }
 
+
 export const singleToArray = <T>(singleOrArray: T | T[]): T[] =>
-  Array.isArray(singleOrArray) ? singleOrArray : [singleOrArray]
+	Array.isArray(singleOrArray) ? singleOrArray : [singleOrArray]
 
 export const filterObject = <T>(
-  obj: { [key: string]: T },
+	obj: { [key: string]: T },
-  filter: (item: [string, T]) => boolean
+	filter: (item: [string, T]) => boolean,
 ): { [key: string]: T } =>
-  Object.fromEntries(Object.entries(obj).filter(filter))
+	Object.fromEntries(Object.entries(obj).filter(filter))
 
 export const filterObjectByKey = <T>(
-  obj: { [key: string]: T },
+	obj: { [key: string]: T },
-  keys: string[]
+	keys: string[],
 ) => filterObject(obj, ([key]) => keys.includes(key))
 
 export const downloadFile = async (url: string, to: string) =>
-  new Promise<void>(async res => {
+	new Promise<void>(async res => {
-    const { data: file } = await axios({
+		const { data: file } = await axios({
-      method: 'get',
+			method: 'get',
-      url: url,
+			url: url,
-      responseType: 'stream',
+			responseType: 'stream',
-    })
+		})
 
-    const stream = createWriteStream(to)
+		const stream = createWriteStream(to)
 
-    const writer = file.pipe(stream)
+		const writer = file.pipe(stream)
-    writer.on('close', () => {
+		writer.on('close', () => {
-      stream.close()
+			stream.close()
-      res()
+			res()
-    })
+		})
-  })
+	})
+
+// Check if is an absolute path, otherwise get the path relative to the config file
+export const pathRelativeToConfigFile = (path: string): string => isAbsolute(path)
+	? path
+	: resolve(dirname(CONFIG_FILE), path)
 
 export const ConfigError = new Error('Config file not found')
+
+export const getFlagsFromLocation = (location: Location, command?: string): string[] => {
+	if (!location.options) return []
+
+	const all = {
+		...location.options.global,
+		...(location.options[command || ''] || {}),
+	}
+
+	let flags: string[] = []
+	// Map the flags to an array for the exec function.
+	for (let [flag, values] of Object.entries(all)) {
+		if (!Array.isArray(values))
+			values = [values]
+
+		for (const value of values)
+			flags = [...flags, `--${String(flag)}`, String(value)]
+	}
+	return flags
+}