v0.9.6: documents, downloads, PDFs and the phone's own files

Four tools and a document writer, plus the storage half of the phone: until
now the picker's answer (a content:// URI) reached the engine as a path, and
it said "cannot read" -- the app had no way to hand a phone's file over.

- md_to_html (new, src/tools/web.c + cst_web_md_document()): markdown in, one
  self-contained HTML document out -- doctype, utf-8, viewport, escaped
  title, a small stylesheet; no script, no external request. It is written
  into the workspace and delivered as a file card (for_user, so it never
  enters a request), the way deliver_file does. The default name comes from
  the source file's stem or, for pasted markdown, from the title with what a
  path cannot carry turned into '-'. It has a preview, so an approval prompt
  says "render x -> y.html" before anything is written.
- download (src/tools/web.c): an http(s) URL into a file in the workspace,
  streamed with a 128 MB ceiling; a redirect chain, the same CA handling and
  the same refusals as web_fetch, but the bytes stay on disk (a half file is
  removed on any failure, including HTTP >= 400). The name comes from the URL
  when the caller names only a directory.
- read_pdf (src/tools/fs_tools.c): the text layer and page count of a PDF,
  which read_file refuses because the bytes are binary -- and a PDF is
  exactly what arrives from a phone. Same extractor an attachment uses, its
  own 4 MiB input ceiling, 256 KiB of text by default with a note when cut.
- session_search (new, src/tools/session_tools.c): every stored session --
  this one and all the ones before it -- searched for a substring, reporting
  the session's id, message count and title, then the matching lines with
  their number and role. Case-insensitive by default, capped at 20 matches.
- device_info (same file, perm_name "device"): version, system/arch/CPU
  count, workspace, sandbox, model and permissions, config/temp/program
  directories, pid -- so the model stops guessing where it runs.
- Tools 3 and 4 come with tests/test_tools_ext.c (also in both build files):
  the document is a whole page, a script tag and a javascript: link do not
  survive, a hostile title stays text and cannot write outside the workspace,
  the preview names the file, read_pdf/download refuse what they must (no
  network in a test), session_search finds a fixture session and says so when
  nothing matches, device_info names the workspace. README's tool table grew
  five rows.

- Android storage: the picker (SAF) answers with a content:// URI, and the
  engine opens files by path, so the app now copies what it points at into
  its own directory first -- app/android/src/xyz/whcself/cstagent/
  AttachFile.java, called through a new app::materializeAttachment()
  (ToolProvider.h: one API for every platform; a desktop strips file:// and
  refuses content:// with a reason). The copy needs no permission at all: the
  grant travels with the URI.
- "All files access" (MANAGE_EXTERNAL_STORAGE) is the other half: it is what
  lets the engine read a path in shared storage that a task names rather than
  picks. Android has no runtime dialog for it, so main.cpp opens the one
  Settings screen once (AttachFile.java's requestAllFilesAccess, with the
  all-apps screen as a fallback) and never insists: without it everything
  works, one picked file at a time. On the NX789J the screen came up, the
  app-op reads back "allow", and a relaunch no longer asks.
- Composer.qml hands a content:// URI through untouched (it is not a path,
  and decoding it would corrupt it); AgentCore::attachFile() turns whatever
  the picker gave into a path first.

Desktop: build clean, make check 36/36 (the new test included, also under
ASan), --uitest all green, --selftest and cst-app-selftest OK.
The APK builds and installs (86 MB, release-signed) with the new Java class
in its dex.

Version 0.9.6.

app/android/AndroidManifest.xml

@@ -18,12 +18,15 @@
 
      Permissions: INTERNET for the providers the user configures,
      POST_NOTIFICATIONS for the app's notification_send tool (Android 13+
-     asks at runtime; the tool stays hidden until it is granted), and the
+     asks at runtime; the tool stays hidden until it is granted), the
      foreground-service permissions with WAKE_LOCK for the service that keeps
-     a turn going while the phone is in a pocket (CstService.java). No
-     storage permission: the file picker is the system one (SAF) and its
-     input is copied into the app's directory. No cleartext traffic: every
-     provider URL is https. -->
+     a turn going while the phone is in a pocket (CstService.java), and
+     MANAGE_EXTERNAL_STORAGE ("all files access", Android 11+) so the engine
+     can read a path in shared storage that a task names rather than picks:
+     Android only offers the Settings screen for it, so main.cpp opens that
+     once and everything still works without it: what the system picker (SAF)
+     hands over is copied into the app's own directory (AttachFile.java). No
+     cleartext traffic: every provider URL is https. -->
 <manifest xmlns:android="http://schemas.android.com/apk/res/android"
     package="xyz.whcself.cstagent"
     android:installLocation="auto"
@@ -38,6 +41,12 @@
     <uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
     <uses-permission android:name="android.permission.FOREGROUND_SERVICE_DATA_SYNC" />
     <uses-permission android:name="android.permission.WAKE_LOCK" />
+    <!-- "All files access": without it the engine can only read the app's own
+         directory, so a task that names a file in shared storage cannot be
+         answered. Android 11+ has no runtime dialog for it, only the Settings
+         screen (AttachFile.java opens it once). Files the user picks work
+         either way: their content:// grant is copied into the app first. -->
+    <uses-permission android:name="android.permission.MANAGE_EXTERNAL_STORAGE" />
 
     <supports-screens
         android:anyDensity="true"

app/android/src/xyz/whcself/cstagent/AttachFile.java

@@ -0,0 +1,189 @@
+package xyz.whcself.cstagent;
+
+import android.content.ActivityNotFoundException;
+import android.content.ContentResolver;
+import android.content.Context;
+import android.content.Intent;
+import android.database.Cursor;
+import android.net.Uri;
+import android.os.Build;
+import android.os.Environment;
+import android.provider.OpenableColumns;
+import android.provider.Settings;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+/* What the user picks in the system file chooser comes back as a content://
+ * URI, and the engine opens files by path -- so the bytes are copied into the
+ * app's own directory first. That copy needs no permission at all (the grant
+ * travels with the URI), which is what makes picking work on a phone that
+ * grants nothing else.
+ *
+ * The "all files access" below is a different thing: it is what lets the
+ * engine read /sdcard by path afterwards (a model that opens a file the user
+ * names rather than picks). Android 11+ has no runtime dialog for it -- only
+ * the Settings screen -- so it is a request, not a demand: everything works
+ * without it, one file at a time.
+ *
+ * Qt swallows a Java exception (it comes back as a null object), so every
+ * method catches its own and answers with a reason instead.
+ */
+public final class AttachFile {
+    private AttachFile() {}
+
+    private static final long MAX_BYTES = 256L * 1024 * 1024;
+
+    public static boolean isContent(String s) {
+        return s != null && s.startsWith("content:");
+    }
+
+    /* content://<uri> -> the path of a copy inside destDir; "" when the string
+     * is already a path; "error: ..." when the copy failed. */
+    public static String materialize(Context ctx, String pathOrUri, String destDir) {
+        if (pathOrUri == null || pathOrUri.isEmpty())
+            return "error: no file was picked";
+        if (!isContent(pathOrUri))
+            return "";
+        try {
+            if (ctx == null)
+                return "error: no Android context";
+            final Uri uri = Uri.parse(pathOrUri);
+            String name = displayName(ctx, uri);
+            if (name == null || name.trim().isEmpty())
+                name = uri.getLastPathSegment();
+            name = sanitize(name);
+            File dir = new File(destDir);
+            if (!dir.isDirectory() && !dir.mkdirs())
+                return "error: cannot use " + destDir;
+            File out = unique(dir, name);
+            final ContentResolver cr = ctx.getContentResolver();
+            final InputStream in = cr.openInputStream(uri);
+            if (in == null)
+                return "error: the picker gave no data for that file";
+            long total = 0;
+            try {
+                final OutputStream os = new FileOutputStream(out);
+                try {
+                    final byte[] buf = new byte[64 * 1024];
+                    int n;
+                    while ((n = in.read(buf)) > 0) {
+                        total += n;
+                        if (total > MAX_BYTES) {
+                            os.close();
+                            in.close();
+                            out.delete();
+                            return "error: that file is larger than 256 MB";
+                        }
+                        os.write(buf, 0, n);
+                    }
+                } finally {
+                    os.close();
+                }
+            } finally {
+                in.close();
+            }
+            if (total == 0) {
+                out.delete();
+                return "error: that file is empty";
+            }
+            return out.getAbsolutePath();
+        } catch (Throwable t) {
+            return "error: cannot read the picked file: " + t;
+        }
+    }
+
+    /* Android 11+ keeps "all files access" in a special app-op; older systems
+     * have no such gate. */
+    public static boolean hasAllFilesAccess(Context ctx) {
+        try {
+            if (Build.VERSION.SDK_INT < 30)
+                return true;
+            return Environment.isExternalStorageManager();
+        } catch (Throwable t) {
+            return false;
+        }
+    }
+
+    /* The Settings screen Android requires for it. "" when it was opened. */
+    public static String requestAllFilesAccess(Context ctx) {
+        try {
+            if (Build.VERSION.SDK_INT < 30 || ctx == null)
+                return "";
+            if (Environment.isExternalStorageManager())
+                return "";
+            final Intent intent = new Intent(
+                    Settings.ACTION_MANAGE_APP_ALL_FILES_ACCESS_PERMISSION,
+                    Uri.parse("package:" + ctx.getPackageName()));
+            intent.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK);
+            try {
+                ctx.startActivity(intent);
+            } catch (ActivityNotFoundException notFound) {
+                final Intent all = new Intent(
+                        Settings.ACTION_MANAGE_ALL_FILES_ACCESS_PERMISSION);
+                all.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK);
+                ctx.startActivity(all);
+            }
+            return "";
+        } catch (Throwable t) {
+            return "cannot open the storage settings: " + t;
+        }
+    }
+
+    private static String displayName(Context ctx, Uri uri) {
+        Cursor c = null;
+        try {
+            c = ctx.getContentResolver().query(uri,
+                    new String[] { OpenableColumns.DISPLAY_NAME }, null, null,
+                    null);
+            if (c != null && c.moveToFirst()) {
+                final int i = c.getColumnIndex(OpenableColumns.DISPLAY_NAME);
+                if (i >= 0)
+                    return c.getString(i);
+            }
+        } catch (Throwable ignored) {
+            /* fall back to the last path segment */
+        } finally {
+            if (c != null)
+                c.close();
+        }
+        return null;
+    }
+
+    /* A plain file name: no separators, no traversal, no control bytes. */
+    private static String sanitize(String name) {
+        if (name == null || name.isEmpty())
+            return "picked.bin";
+        final StringBuilder b = new StringBuilder();
+        for (int i = 0; i < name.length() && b.length() < 120; i++) {
+            final char ch = name.charAt(i);
+            if (ch == '/' || ch == '\\' || ch < 0x20)
+                b.append('_');
+            else
+                b.append(ch);
+        }
+        String s = b.toString();
+        while (s.startsWith("."))
+            s = s.substring(1);
+        if (s.isEmpty())
+            return "picked.bin";
+        return s;
+    }
+
+    private static File unique(File dir, String name) {
+        File f = new File(dir, name);
+        if (!f.exists())
+            return f;
+        final int dot = name.lastIndexOf('.');
+        final String stem = dot > 0 ? name.substring(0, dot) : name;
+        final String ext = dot > 0 ? name.substring(dot) : "";
+        for (int i = 2; i < 1000; i++) {
+            f = new File(dir, stem + "-" + i + ext);
+            if (!f.exists())
+                return f;
+        }
+        return f;
+    }
+}

app/qml/Composer.qml

@@ -31,6 +31,12 @@ Rectangle {
     /* A file: from the picker, or a URL dropped on the window. */
     function localPath(url) {
         let s = String(url)
+        // A phone's picker (SAF) hands over content:// URIs: they are not
+        // paths and must not be decoded -- the app copies what they point at
+        // into its own directory before the engine sees it (see
+        // app::materializeAttachment).
+        if (s.startsWith("content://"))
+            return s
         if (s.startsWith("file://"))
             s = s.substring(7)
         /* Windows keeps its drive letter after the slash. */

app/src/core/AgentCore.cpp

@@ -1034,11 +1034,19 @@ QString AgentCore::attachFile(const QString &path) {
     return QStringLiteral("no engine");
   if (path.isEmpty())
     return QStringLiteral("no file");
+  /* What a picker hands over is not always a path: on a phone it is a SAF
+   * content:// URI, and the app copies that into its own directory first (the
+   * engine opens files by path). A desktop passes the path through. */
+  QString err2;
+  const QString local = app::materializeAttachment(path, &err2);
+  if (local.isEmpty())
+    return err2.isEmpty() ? QStringLiteral("that file cannot be attached")
+                          : err2;
   char err[256] = "";
-  if (cst_agent_attach_file(m_agent, path.toUtf8().constData(), err,
+  if (cst_agent_attach_file(m_agent, local.toUtf8().constData(), err,
                             sizeof(err)) != 0)
     return utf8(err[0] ? err : "that file cannot be attached");
-  m_attachNames << QFileInfo(path).fileName();
+  m_attachNames << QFileInfo(local).fileName();
   emit pendingChanged();
   return QString();
 }

app/src/core/ToolProvider.h

@@ -78,6 +78,22 @@ QString holdForeground(const QString &text);
 /* Drop it again, at the end of the turn (a phone: the notification goes). */
 void releaseForeground();
 
+/* Turn what a file chooser handed over into a path the engine can open. A
+ * desktop picker gives a path and nothing happens; a phone's is a SAF
+ * content:// URI, which the app copies into its own directory first (the
+ * grant travels with the URI, so no permission is involved -- and without
+ * the copy the engine, which opens files by path, would fail with "cannot
+ * read"). Returns the path to read, or "" with *err filled. */
+QString materializeAttachment(const QString &pathOrUri, QString *err);
+/* Does this app have "all files access" (Android 11+)? A desktop always
+ * does. Without it the app still attaches whatever the picker handed it; what
+ * it cannot do is read shared storage by path. */
+bool hasAllFilesAccess();
+/* Ask for it, through the one screen Android offers (there is no runtime
+ * dialog for it). A desktop, a phone that already has it or one too old to
+ * know about it returns "" and nothing opens. */
+QString requestAllFilesAccess();
+
 }  // namespace app
 
 #endif /* CST_APP_TOOLPROVIDER_H */

app/src/main.cpp

@@ -257,6 +257,26 @@ int main(int argc, char **argv) {
     QtAndroidPrivate::requestPermission(
         QStringLiteral("android.permission.POST_NOTIFICATIONS"));
   }
+  /* "All files access" is the other half of storage: it is what lets the
+   * engine read a path in shared storage (a file the user names, not one the
+   * picker hands over -- that one is copied into the app either way, see
+   * AttachFile.java). Android has no runtime dialog for it, only the Settings
+   * screen, so it is opened once (a marker file remembers) and never
+   * insisted on: without it everything still works, one picked file at a
+   * time. The line below is what logcat shows when a device behaves oddly. */
+  const bool allFiles = app::hasAllFilesAccess();
+  qInfo("storage: all files access = %s", allFiles ? "yes" : "no");
+  if (!allFiles) {
+    const QString marker = androidRoot() + QStringLiteral("/.all-files-asked");
+    if (!QFileInfo::exists(marker)) {
+      const QString why = app::requestAllFilesAccess();
+      if (!why.isEmpty())
+        qWarning("all files access: %s", qUtf8Printable(why));
+      QFile asked(marker);
+      if (asked.open(QIODevice::WriteOnly))
+        asked.write("1\n");
+    }
+  }
 #endif
   if (parser.isSet(planOpt))
     core.setPlanMode(true);

app/src/platform/android/AndroidTools.cpp

@@ -27,6 +27,7 @@
 #include <QGuiApplication>
 #include <QJniObject>
 #include <QMetaObject>
+#include <QStandardPaths>
 #include <QString>
 #include <QStringList>
 #include <QThread>
@@ -662,6 +663,89 @@ void releaseForeground() {
   });
 }
 
+/* What the picker gave us -> a path the engine can open. A SAF URI is copied
+ * into the app's own cache (the URI's grant covers the read; no permission is
+ * involved), a plain path is left alone. */
+QString materializeAttachment(const QString &pathOrUri, QString *err) {
+  if (err)
+    err->clear();
+  if (pathOrUri.isEmpty())
+    return QString();
+  if (!pathOrUri.startsWith(QLatin1String("content://"))) {
+    QString p = pathOrUri;
+    if (p.startsWith(QLatin1String("file://")))
+      p = p.mid(7);
+    return p;
+  }
+  const QString dest = QStandardPaths::writableLocation(
+                           QStandardPaths::CacheLocation) +
+                       QStringLiteral("/picked");
+  QString result, failure;
+  onGuiThread([&] {
+    QJniObject ctx = appContext();
+    if (!ctx.isValid()) {
+      failure = QStringLiteral("no Android context to copy the file with");
+      return;
+    }
+    QJniObject out = QJniObject::callStaticObjectMethod(
+        "xyz/whcself/cstagent/AttachFile", "materialize",
+        "(Landroid/content/Context;Ljava/lang/String;Ljava/lang/String;)"
+        "Ljava/lang/String;",
+        ctx.object<jobject>(), QJniObject::fromString(pathOrUri).object<jstring>(),
+        QJniObject::fromString(dest).object<jstring>());
+    if (!out.isValid()) {
+      failure = QStringLiteral("the phone could not hand that file over");
+      return;
+    }
+    const QString s = out.toString();
+    if (s.startsWith(QLatin1String("error: ")))
+      failure = s.mid(7);
+    else if (!s.isEmpty())
+      result = s;
+    /* "" means it was a path all along; the caller keeps what it had */
+  });
+  if (!failure.isEmpty()) {
+    if (err)
+      *err = failure;
+    return QString();
+  }
+  return result.isEmpty() ? pathOrUri : result;
+}
+
+bool hasAllFilesAccess() {
+  bool granted = true;
+  onGuiThread([&] {
+    QJniObject ctx = appContext();
+    if (!ctx.isValid())
+      return;
+    granted = QJniObject::callStaticMethod<jboolean>(
+        "xyz/whcself/cstagent/AttachFile", "hasAllFilesAccess",
+        "(Landroid/content/Context;)Z", ctx.object<jobject>());
+  });
+  return granted;
+}
+
+QString requestAllFilesAccess() {
+  if (qEnvironmentVariableIsSet("CST_APP_NO_PERM"))
+    return QString(); /* a test run touches no settings */
+  QString err;
+  onGuiThread([&] {
+    QJniObject ctx = appContext();
+    if (!ctx.isValid())
+      return;
+    QJniObject out = QJniObject::callStaticObjectMethod(
+        "xyz/whcself/cstagent/AttachFile", "requestAllFilesAccess",
+        "(Landroid/content/Context;)Ljava/lang/String;",
+        ctx.object<jobject>());
+    if (out.isValid()) {
+      const QString s = out.toString();
+      if (!s.isEmpty())
+        err = s;
+    }
+  });
+  return err;
+}
+
 int platformRegisterAndroid(AgentCore *core) {
   if (!core)
     return 0;

app/src/platform/linux/NotifyTool.cpp

@@ -134,9 +134,29 @@ QString holdForeground(const QString &text) {
   (void)text;
   return QString();
 }
-
 void releaseForeground() {}
 
+/* A desktop picker hands over a path, so there is nothing to turn into one:
+ * a file:// URL loses its scheme and a content:// URI (a portal that never
+ * gave us a path) is refused with what the user should do instead. */
+QString materializeAttachment(const QString &pathOrUri, QString *err) {
+  QString s = pathOrUri;
+  if (s.startsWith(QLatin1String("file://")))
+    s = s.mid(7);
+  if (s.startsWith(QLatin1String("content://"))) {
+    if (err)
+      *err = QStringLiteral("that picker handed over a content:// URI this "
+                            "desktop cannot open; pick the file by path "
+                            "instead");
+    return QString();
+  }
+  return s;
+}
+
+/* Every desktop process reads the files its user can. */
+bool hasAllFilesAccess() { return true; }
+QString requestAllFilesAccess() { return QString(); }
+
 int platformRegisterLinux(AgentCore *core) {
   static const cst_tool_param_t params[CST_TOOL_MAX_PARAMS] = {
       {"message", "string", 1, "text of the notification"},