v0.9.0: the Qt front end as an Android APK, images in the transcript
Two strands land together: the window draws pictures now, and the whole front end builds, installs and runs on the phone with the engine linked in-process. Images (the "make a picture actually show up" part) - Engine, additively: cst_agent_hooks_t.on_tool_images(call, list, n, ud) fires when a call left attachments behind, and cst_agent_pending_rel() / cst_agent_pending_is_image() read that list without the app touching the store. tests/test_attach.c::test_agent_pending pins the three cases (an image, a stored name that passes cst_attach_rel_ok, out-of-range NULL). - App: image://cstattach/<stored name> (app/src/ui/SessionImages.cpp) serves only the current session's attachment directory, checked with cst_attach_rel_ok, decodes down to the requested sourceSize and never upscales, and holds a mutex because requestImage() runs on the render thread while the directory can change underneath it. AgentCore gained attachmentDir(), pendingImages(), the toolImages()/promptImages() signals and hookToolImages(); history() gives replayed rows their images too. - QML: ImageStrip.qml (a thumb strip, <=150 px tall, shaped by the picture, tooltip with name/size/origin, the file's name when it went missing), wired into MessageCard and ToolCard. Qt 6's ListModel cannot hold a list-valued role (append() reads back undefined, setProperty drops it), so a row carries one image per line as <stored>|<name>|<origin>|<w>x<h> and ImageStrip decodes that. Android (the engine is already in the APK; this is the front end) - app/src/main.cpp: on Android the workspace, config, tmp and HOME go under the app's own data directory before the engine opens anything, because QDir::currentPath() is "/" there and /tmp is not writable. - app/android/: the manifest (QtApplication/QtActivity, INTERNET and POST_NOTIFICATIONS, adjustResize, no storage permission -- the picker will be the system one), strings, a vector icon; minSdk 24 and targetSdk 35 come from app/CMakeLists.txt through QT_ANDROID_MIN/TARGET_SDK_VERSION. - app/src/platform/android/AndroidTools.cpp: the Android file behind PlatformInit(), owner "app:android" (notification, clipboard, share sheet, file picker are stubs for now, so the tool list does not lie about them). - app/tools/selftest.c decides its scratch workspace with os_temp_dir(), so the C smoke test runs on the device too. - scripts/build_apk_android.sh: sources to installed APK in one command (environment checks, qt-cmake configure, --target apk, adb install), and the host-side plumbing the NixOS + qt.io combination needs -- the NDK sysroot include dirs for Qt's FindGLESv2/FindEGL (they only look under <sysroot>/include), -L$PREFIX/lib on the shared/module link lines as well as the executables', the Android debug keystore created on first use with QT_ANDROID_SIGN_APK + QT_ANDROID_KEYSTORE_* (androiddeployqt copies the unsigned gradle output and signs it itself), and the #!/bin/bash shebang of the build-tools wrappers, which NixOS does not have. - scripts/gradle-android-init.gradle (installed into ~/.gradle/init.d/): the release lint task is switched off -- lintVitalAnalyzeRelease resolves its own tool jars straight from dl.google.com, bypassing every repository the build configures, and fails the build after all the real work is done. - Icons: app/icons/*.svg (Breeze's 24@3x files, the 19 names the window asks for) are compiled in as :/cst-icons and ThemeIcons.cpp falls back to them when the platform has no icon theme. On the phone the Qt theme lookup found nothing and the QML glyph fallback was an empty box in the phone's fonts. - docs/android.md: "The APK (Qt front end)" -- the SDK assembled without sdkmanager, the host Qt's missing libraries on NixOS, the Gradle mirrors, lint, signing -- and a Status that says what runs where. app/README.md gains the images and APK lines, and stops claiming Android platform tools that are still stubs. Verified: engine `make check` green; desktop `cst-app-selftest: OK` and --uitest all checks pass; phone (NX789J, Android 16, arm64-v8a) -- the APK installs, starts on the workspace under /data/user/0/xyz.whcself.cstagent/ files/, a turn reaches a configured provider over HTTPS from inside the ART process and renders the reply, rotation keeps the session and docks the sidebar, and the toolbar/card icons draw. Version 0.9.0. mode = coding
This commit is contained in:
@@ -17,6 +17,7 @@
|
||||
#include <cstdlib>
|
||||
|
||||
#include "cst_agent.h"
|
||||
#include "cst_attach.h"
|
||||
#include "cst_models.h"
|
||||
#include "cst_provider_switch.h"
|
||||
#include "cst_web_md.h"
|
||||
@@ -53,6 +54,7 @@ AgentCore::AgentCore(QObject *parent) : QObject(parent) {
|
||||
m_hooks.on_think_delta = hookThink;
|
||||
m_hooks.on_tool_event = hookToolEvent;
|
||||
m_hooks.on_tool_result = hookToolResult;
|
||||
m_hooks.on_tool_images = hookToolImages;
|
||||
m_hooks.on_todo = hookTodo;
|
||||
m_hooks.on_permission = hookPermission;
|
||||
m_hooks.on_ask_user = hookAskUser;
|
||||
@@ -172,7 +174,12 @@ void AgentCore::close() {
|
||||
void AgentCore::send(const QString &prompt) {
|
||||
if (!m_agent || m_running || prompt.trimmed().isEmpty())
|
||||
return;
|
||||
/* The prompt takes the pending images with it: the view shows them on the
|
||||
* message the composer just appended, before the turn can say anything. */
|
||||
const QString sending = pendingImages();
|
||||
setRunning(true);
|
||||
if (!sending.isEmpty())
|
||||
emit promptImages(sending);
|
||||
m_turn = new Turn(this, prompt);
|
||||
m_turn->start();
|
||||
}
|
||||
@@ -314,9 +321,41 @@ QVariantMap sessionRow(const QString &kind, const QString &role) {
|
||||
r["detail"] = QString();
|
||||
r["output"] = QString();
|
||||
r["exitCode"] = 0;
|
||||
r["images"] = QString();
|
||||
return r;
|
||||
}
|
||||
|
||||
/* One field of one picture line: names come from the user and from tool
|
||||
* output, so a separator may not split a line in two. */
|
||||
QString field(const char *s) {
|
||||
QString out = utf8(s);
|
||||
out.replace(QLatin1Char('|'), QLatin1Char('/'));
|
||||
out.replace(QLatin1Char('\n'), QLatin1Char(' '));
|
||||
return out;
|
||||
}
|
||||
|
||||
/* The images of one message or tool result, one line per picture:
|
||||
*
|
||||
* <stored name>|<name>|<origin>|<w>x<h>
|
||||
*
|
||||
* A list role is not an option: a ListModel keeps what fits in a QVariant of
|
||||
* a simple type -- a list reads back as undefined there, and setProperty drops
|
||||
* the role -- so the rows carry this text and ImageStrip.qml decodes it. A
|
||||
* document (PDF, text) is not a picture and stays out of it. */
|
||||
QString imagesText(const cst_attachment_t *list, int n) {
|
||||
QStringList lines;
|
||||
for (int i = 0; list && i < n; i++) {
|
||||
const cst_attachment_t *a = &list[i];
|
||||
if (!cst_attach_is_image(a))
|
||||
continue;
|
||||
lines << QStringLiteral("%1|%2|%3|%4x%5")
|
||||
.arg(field(a->rel), field(a->name), field(a->origin))
|
||||
.arg(a->width)
|
||||
.arg(a->height);
|
||||
}
|
||||
return lines.join(QLatin1Char('\n'));
|
||||
}
|
||||
|
||||
/* The open tool row this result belongs to, -1 when none. Rows are paired by
|
||||
* call id first: the same tool may run twice in a turn, and pairing by name
|
||||
* would hang the first card on "running" forever. Only a call without an id
|
||||
@@ -354,6 +393,20 @@ QString AgentCore::sessionId() const {
|
||||
return utf8(id);
|
||||
}
|
||||
|
||||
/* Where the session keeps its images: what image://cstattach/ serves, and
|
||||
* what the view asks for when the session changes. */
|
||||
QString AgentCore::attachmentDir() const {
|
||||
if (!m_agent)
|
||||
return QString();
|
||||
const char *sess = cst_agent_session_path(m_agent);
|
||||
if (!sess || !sess[0])
|
||||
return QString();
|
||||
char dir[4096] = "";
|
||||
if (cst_attach_dir(sess, dir, sizeof(dir)) != 0)
|
||||
return QString();
|
||||
return utf8(dir);
|
||||
}
|
||||
|
||||
QVariantList AgentCore::sessions() const {
|
||||
QVariantList out;
|
||||
if (m_workspace.isEmpty())
|
||||
@@ -404,12 +457,14 @@ QVariantList AgentCore::history() const {
|
||||
r["name"] = name;
|
||||
r["callId"] = callId;
|
||||
r["output"] = utf8(m->content);
|
||||
r["images"] = imagesText(m->attach, m->nattach);
|
||||
r["state"] = "done";
|
||||
r["complete"] = true;
|
||||
rows.append(r);
|
||||
} else {
|
||||
QVariantMap r = rows.at(at).toMap();
|
||||
r["output"] = utf8(m->content);
|
||||
r["images"] = imagesText(m->attach, m->nattach);
|
||||
r["state"] = "done";
|
||||
r["complete"] = true;
|
||||
rows[at] = r;
|
||||
@@ -426,13 +481,16 @@ QVariantList AgentCore::history() const {
|
||||
if (m->role == CST_ROLE_USER) {
|
||||
QVariantMap r = sessionRow(QStringLiteral("message"), QStringLiteral("user"));
|
||||
QString text = utf8(m->content);
|
||||
if (m->nattach > 0) {
|
||||
QStringList names;
|
||||
for (int a = 0; a < m->nattach; a++)
|
||||
/* A document keeps its name in the text; a picture is shown instead
|
||||
* (the images below), so only the rest is worth naming. */
|
||||
QStringList names;
|
||||
for (int a = 0; a < m->nattach; a++)
|
||||
if (!cst_attach_is_image(&m->attach[a]))
|
||||
names << utf8(m->attach[a].name);
|
||||
if (!names.isEmpty())
|
||||
text += QStringLiteral("\n\n[attached: %1]").arg(names.join(", "));
|
||||
}
|
||||
r["text"] = text;
|
||||
r["images"] = imagesText(m->attach, m->nattach);
|
||||
r["complete"] = true;
|
||||
rows.append(r);
|
||||
continue;
|
||||
@@ -802,6 +860,30 @@ QVariantMap AgentCore::usage() const {
|
||||
|
||||
/* --- attachments ---------------------------------------------------------- */
|
||||
|
||||
/* What the prompt about to be sent carries, ready for the view. The stored
|
||||
* name is the engine's (the file already lives next to the session), the
|
||||
* display name is the one the window kept when it copied the file in. */
|
||||
QString AgentCore::pendingImages() const {
|
||||
QStringList lines;
|
||||
if (!m_agent)
|
||||
return QString();
|
||||
const int n = cst_agent_pending_count(m_agent);
|
||||
for (int i = 0; i < n; i++) {
|
||||
if (!cst_agent_pending_is_image(m_agent, i))
|
||||
continue;
|
||||
const char *rel = cst_agent_pending_rel(m_agent, i);
|
||||
if (!rel || !rel[0])
|
||||
continue;
|
||||
const QString name =
|
||||
i < m_attachNames.size() ? m_attachNames.at(i) : utf8(rel);
|
||||
/* The size is not known before the engine stores the file; the strip
|
||||
* falls back to a picture's usual shape until it is. */
|
||||
lines << QStringLiteral("%1|%2|user|0x0")
|
||||
.arg(field(rel), field(name.toUtf8().constData()));
|
||||
}
|
||||
return lines.join(QLatin1Char('\n'));
|
||||
}
|
||||
|
||||
QStringList AgentCore::pendingAttachments() const {
|
||||
QStringList out;
|
||||
if (!m_agent)
|
||||
@@ -937,6 +1019,16 @@ void AgentCore::hookToolResult(const ctl_tool_call_t *call, const char *out,
|
||||
[self, id, n, exit_code, o] { emit self->toolResult(id, n, exit_code, o); });
|
||||
}
|
||||
|
||||
void AgentCore::hookToolImages(const ctl_tool_call_t *call,
|
||||
const cst_attachment_t *list, int n, void *ud) {
|
||||
AgentCore *self = static_cast<AgentCore *>(ud);
|
||||
const QString images = imagesText(list, n);
|
||||
if (images.isEmpty())
|
||||
return; /* a call that made no picture needs no signal */
|
||||
const QString id = QString::fromUtf8(call && call->id ? call->id : "");
|
||||
self->post([self, id, images] { emit self->toolImages(id, images); });
|
||||
}
|
||||
|
||||
void AgentCore::hookTodo(const char *text, void *ud) {
|
||||
AgentCore *self = static_cast<AgentCore *>(ud);
|
||||
const QString t = QString::fromUtf8(text ? text : "");
|
||||
|
||||
@@ -63,7 +63,8 @@ class AgentCore : public QObject {
|
||||
Q_INVOKABLE QVariantList sessions() const;
|
||||
/* The open session as transcript rows, in exactly the shape the live
|
||||
* signals build (kind, role, text, name, state, detail, output, exitCode,
|
||||
* complete), so a reopened session renders like the turn that wrote it. */
|
||||
* complete, images), so a reopened session renders like the turn that wrote
|
||||
* it. */
|
||||
Q_INVOKABLE QVariantList history() const;
|
||||
/* None of these run during a turn; all return false when one does. */
|
||||
Q_INVOKABLE bool newSession();
|
||||
@@ -128,6 +129,16 @@ class AgentCore : public QObject {
|
||||
* clipboard) or stay with the editor (text). */
|
||||
Q_INVOKABLE bool clipboardHasImage() const;
|
||||
Q_INVOKABLE void clearPending();
|
||||
/* The images among the pending attachments, in the engine's order, in the
|
||||
* same text history() and toolImages() carry (see imagesText()): one line
|
||||
* per picture, "<stored name>|<name>|<origin>|<w>x<h>". Empty for a PDF or
|
||||
* a text file, and for a picture whose stored name the engine did not
|
||||
* report. */
|
||||
Q_INVOKABLE QString pendingImages() const;
|
||||
/* Where the current session keeps its stored images: the directory
|
||||
* image://cstattach/ serves (SessionImages.cpp). "" until the engine is
|
||||
* open. It follows the session. */
|
||||
Q_INVOKABLE QString attachmentDir() const;
|
||||
/* 1 the model is shown images, 0 it is not (they go as notes), -1 unknown. */
|
||||
Q_INVOKABLE int seesImages() const;
|
||||
|
||||
@@ -176,6 +187,12 @@ class AgentCore : public QObject {
|
||||
const QString &detail);
|
||||
void toolResult(const QString &id, const QString &name, int exitCode,
|
||||
const QString &output);
|
||||
/* Images a tool produced (view_image, an MCP image), right after its
|
||||
* result, as the text ImageStrip.qml decodes. */
|
||||
void toolImages(const QString &callId, const QString &images);
|
||||
/* The images the prompt that was just sent carries, in the same shape,
|
||||
* fired before the turn's first delta. */
|
||||
void promptImages(const QString &images);
|
||||
void todoChanged(const QString &text);
|
||||
void permissionRequested(const QString &tool, const QString &detail,
|
||||
const QString &preview);
|
||||
@@ -203,6 +220,8 @@ class AgentCore : public QObject {
|
||||
const char *detail, void *ud);
|
||||
static void hookToolResult(const ctl_tool_call_t *call, const char *out,
|
||||
size_t len, int exit_code, void *ud);
|
||||
static void hookToolImages(const ctl_tool_call_t *call,
|
||||
const cst_attachment_t *list, int n, void *ud);
|
||||
static void hookTodo(const char *text, void *ud);
|
||||
static int hookPermission(const char *tool, const char *detail,
|
||||
const char *preview, void *ud);
|
||||
|
||||
+39
-1
@@ -27,9 +27,24 @@
|
||||
#include "core/AgentCore.h"
|
||||
#include "core/ToolProvider.h"
|
||||
#include "ui/UiTest.h"
|
||||
#include "ui/SessionImages.h"
|
||||
#include "ui/ThemeIcons.h"
|
||||
#include "cst_version.h"
|
||||
|
||||
#if defined(Q_OS_ANDROID)
|
||||
/* Where an Android app is allowed to write: its own data directory. There is
|
||||
* no usable $HOME, no writable "/" (QDir::currentPath()) and no /tmp, while
|
||||
* the engine wants a config dir ($XDG_CONFIG_HOME/cst-agent, os_config_dir())
|
||||
* and a scratch dir (the first writable entry of os_temp_dir()'s list). All
|
||||
* of them are pointed here -- once, before the engine exists. */
|
||||
QString androidRoot() {
|
||||
const QString root =
|
||||
QStandardPaths::writableLocation(QStandardPaths::AppDataLocation);
|
||||
return root.isEmpty() ? QDir::tempPath() + QStringLiteral("/cst-agent")
|
||||
: root;
|
||||
}
|
||||
#endif
|
||||
|
||||
namespace {
|
||||
|
||||
int runSelftest() {
|
||||
@@ -141,8 +156,13 @@ int main(int argc, char **argv) {
|
||||
parser.process(app);
|
||||
|
||||
QString ws = parser.value(wsOpt);
|
||||
if (ws.isEmpty())
|
||||
if (ws.isEmpty()) {
|
||||
#if defined(Q_OS_ANDROID)
|
||||
ws = androidRoot() + QStringLiteral("/workspace");
|
||||
#else
|
||||
ws = QDir::currentPath();
|
||||
#endif
|
||||
}
|
||||
ws = QFileInfo(ws).absoluteFilePath();
|
||||
|
||||
/* --uitest drives a scratch workspace when none was given: the session
|
||||
@@ -160,6 +180,17 @@ int main(int argc, char **argv) {
|
||||
}
|
||||
}
|
||||
|
||||
#if defined(Q_OS_ANDROID)
|
||||
/* Before the engine is created: its settings, credentials and session store
|
||||
* all resolve their paths from the environment. */
|
||||
const QString root = androidRoot();
|
||||
for (const QString &sub : {QStringLiteral("config"), QStringLiteral("tmp")})
|
||||
QDir().mkpath(root + QLatin1Char('/') + sub);
|
||||
qputenv("XDG_CONFIG_HOME", (root + QStringLiteral("/config")).toUtf8());
|
||||
qputenv("TMPDIR", (root + QStringLiteral("/tmp")).toUtf8());
|
||||
qputenv("HOME", root.toUtf8());
|
||||
#endif
|
||||
|
||||
AgentCore core;
|
||||
QString err;
|
||||
if (!core.open(ws, parser.value(providerOpt), parser.value(modelOpt),
|
||||
@@ -195,6 +226,13 @@ int main(int argc, char **argv) {
|
||||
QQmlApplicationEngine engine;
|
||||
/* image://csttheme/ plus the desktop's icon theme behind it. */
|
||||
app::installThemeIcons(&engine);
|
||||
/* image://cstattach/: the session's own store, so the pictures a tool made
|
||||
* (view_image, an MCP image) and the ones the user attached are shown, not
|
||||
* only named. The directory follows the session. */
|
||||
app::SessionImages *images = app::installSessionImages(&engine);
|
||||
images->setDir(core.attachmentDir());
|
||||
QObject::connect(&core, &AgentCore::sessionChanged, images,
|
||||
[images, &core] { images->setDir(core.attachmentDir()); });
|
||||
engine.rootContext()->setContextProperty(QStringLiteral("agent"), &core);
|
||||
engine.rootContext()->setContextProperty(QStringLiteral("initialWidth"), winW);
|
||||
engine.rootContext()->setContextProperty(QStringLiteral("initialHeight"), winH);
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
/* Platform tools on Android, implemented by the app: the same contract as
|
||||
* app/src/platform/linux/NotifyTool.cpp, with Android APIs behind it.
|
||||
*
|
||||
* The engine only ever sees the capability: `platforms = kAndroidOnly` (the
|
||||
* engine reports "android" when it was compiled with the NDK, see
|
||||
* src/tools/registry.c: cst_platform_name()) plus an `available` probe decide
|
||||
* whether a tool reaches the prompt and tools[] at all -- a Termux build on
|
||||
* the same device reports "termux" and gets the android tools too, an APK
|
||||
* never gets the desktop ones. `owner = "app:android"` is what
|
||||
* unregisterPlatformTools() drops again.
|
||||
*
|
||||
* WHERE THE CODE RUNS: `run_ex` is called on the turn's worker thread, never
|
||||
* on the GUI thread. JNI calls are fine from any thread (QJniEnvironment
|
||||
* attaches it), Qt GUI objects are not -- a tool that needs the widget tree
|
||||
* has to hop back with a blocking queued call.
|
||||
*
|
||||
* WHAT GOES HERE (one at a time, each with its own probe):
|
||||
* notification_send NotificationChannel + NotificationManager through
|
||||
* JNI; hidden until POST_NOTIFICATIONS is granted
|
||||
* (Android 13+), which the UI asks for on first use.
|
||||
* clipboard_get/set the phone clipboard through Qt's QClipboard, taking
|
||||
* the name of the desktop tool that cannot work here
|
||||
* (see app/README.md: a front end may shadow a built-in
|
||||
* that has no working backend on its host).
|
||||
* share ACTION_SEND of a file or text through the system share
|
||||
* sheet -- the phone's answer to "show this to the user".
|
||||
*
|
||||
* Nothing is registered yet: the first APK ships the engine, the window and
|
||||
* the tool contract, and the tools above arrive next (docs/android.md). */
|
||||
#include <QString>
|
||||
|
||||
#include "core/AgentCore.h"
|
||||
#include "core/ToolProvider.h"
|
||||
|
||||
namespace app {
|
||||
|
||||
int platformRegisterAndroid(AgentCore *core) {
|
||||
if (!core)
|
||||
return 0;
|
||||
return 0; /* no Android tools yet */
|
||||
}
|
||||
|
||||
} // namespace app
|
||||
@@ -0,0 +1,81 @@
|
||||
#include "ui/SessionImages.h"
|
||||
|
||||
#include <QFileInfo>
|
||||
#include <QImageReader>
|
||||
#include <QQmlApplicationEngine>
|
||||
|
||||
#include "cst_attach.h"
|
||||
|
||||
namespace app {
|
||||
namespace {
|
||||
|
||||
/* "img-1a2b3c4d.png?w=160": the stored name, then what the view asked for.
|
||||
* The size comes from the QML side as sourceSize, which Qt passes in as
|
||||
* requestedSize; the query string is accepted too, so a plain URL can ask. */
|
||||
QString relOf(const QString &id) {
|
||||
const int q = id.indexOf(QLatin1Char('?'));
|
||||
return q < 0 ? id : id.left(q);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
SessionImages::SessionImages()
|
||||
: QQuickImageProvider(QQuickImageProvider::Image) {}
|
||||
|
||||
void SessionImages::setDir(const QString &dir) {
|
||||
QMutexLocker lock(&m_mu);
|
||||
m_dir = dir;
|
||||
}
|
||||
|
||||
QImage SessionImages::requestImage(const QString &id, QSize *size,
|
||||
const QSize &requestedSize) {
|
||||
const QString rel = relOf(id);
|
||||
if (!cst_attach_rel_ok(rel.toUtf8().constData()))
|
||||
return QImage(); /* QML sees Image.Error and falls back to the name */
|
||||
QString dir;
|
||||
{
|
||||
QMutexLocker lock(&m_mu);
|
||||
dir = m_dir;
|
||||
}
|
||||
if (dir.isEmpty())
|
||||
return QImage();
|
||||
const QString path = dir + QLatin1Char('/') + rel;
|
||||
if (!QFileInfo::exists(path))
|
||||
return QImage();
|
||||
|
||||
QImageReader reader(path);
|
||||
reader.setAutoTransform(true);
|
||||
const QSize full = reader.size();
|
||||
if (!full.isValid()) {
|
||||
/* No size in the header (or not a picture at all): hand back what Qt
|
||||
* makes of it and let the view scale it. */
|
||||
const QImage img = reader.read();
|
||||
if (size)
|
||||
*size = img.size();
|
||||
return img;
|
||||
}
|
||||
/* Shrink to what the view shows, never up: a screenshot is decoded at the
|
||||
* size of its thumbnail instead of at 4K. */
|
||||
const bool smaller = requestedSize.isValid() && requestedSize.width() > 0 &&
|
||||
requestedSize.height() > 0 &&
|
||||
(full.width() > requestedSize.width() ||
|
||||
full.height() > requestedSize.height());
|
||||
if (smaller) {
|
||||
const QSize scaled = full.scaled(requestedSize, Qt::KeepAspectRatio);
|
||||
reader.setScaledSize(scaled);
|
||||
if (size)
|
||||
*size = scaled;
|
||||
} else if (size) {
|
||||
*size = full;
|
||||
}
|
||||
return reader.read();
|
||||
}
|
||||
|
||||
SessionImages *installSessionImages(QQmlApplicationEngine *engine) {
|
||||
SessionImages *images = new SessionImages;
|
||||
if (engine)
|
||||
engine->addImageProvider(QStringLiteral("cstattach"), images);
|
||||
return images;
|
||||
}
|
||||
|
||||
} // namespace app
|
||||
@@ -0,0 +1,45 @@
|
||||
/* The session's stored images as a QML image provider:
|
||||
*
|
||||
* image://cstattach/<stored name> (see cst_attach.h: "img-1a2b3c4d.png")
|
||||
*
|
||||
* The files live in the current session's attachment directory, which is what
|
||||
* AgentCore::attachmentDir() names. Only that directory is served, and only
|
||||
* names that pass the engine's own cst_attach_rel_ok() -- a transcript can
|
||||
* carry hand-edited JSON, and nothing in it may reach outside the store.
|
||||
*
|
||||
* requestImage() runs on Quick's render thread while setDir() comes from the
|
||||
* window's thread, hence the mutex. */
|
||||
#ifndef CST_APP_SESSION_IMAGES_H
|
||||
#define CST_APP_SESSION_IMAGES_H
|
||||
|
||||
#include <QImage>
|
||||
#include <QMutex>
|
||||
#include <QQuickImageProvider>
|
||||
#include <QString>
|
||||
|
||||
class QQmlApplicationEngine;
|
||||
|
||||
namespace app {
|
||||
|
||||
class SessionImages : public QQuickImageProvider {
|
||||
public:
|
||||
SessionImages();
|
||||
|
||||
/* The directory holding the current session's images; "" disables serving. */
|
||||
void setDir(const QString &dir);
|
||||
|
||||
QImage requestImage(const QString &id, QSize *size,
|
||||
const QSize &requestedSize) override;
|
||||
|
||||
private:
|
||||
mutable QMutex m_mu;
|
||||
QString m_dir;
|
||||
};
|
||||
|
||||
/* Registers image://cstattach/ on the engine and hands the provider back, so
|
||||
* the caller can keep its directory in step with the agent's session. */
|
||||
SessionImages *installSessionImages(QQmlApplicationEngine *engine);
|
||||
|
||||
} // namespace app
|
||||
|
||||
#endif // CST_APP_SESSION_IMAGES_H
|
||||
@@ -48,6 +48,21 @@ void adoptIconTheme() {
|
||||
}
|
||||
}
|
||||
|
||||
/* The bundled set: app/icons/*.svg, compiled in as :/cst-icons/<name>.svg by
|
||||
* app/CMakeLists.txt. A machine without an icon theme -- Android, where the
|
||||
* Qt theme lookup finds nothing and QML was left showing its glyph fallback,
|
||||
* which the phone fonts cannot draw -- gets the same icons from here. */
|
||||
QIcon bundledIcon(const QString &name) {
|
||||
return QIcon(QStringLiteral(":/cst-icons/") + name + QStringLiteral(".svg"));
|
||||
}
|
||||
|
||||
QIcon iconForName(const QString &name) {
|
||||
const QIcon themed = QIcon::fromTheme(name);
|
||||
if (!themed.isNull())
|
||||
return themed;
|
||||
return bundledIcon(name);
|
||||
}
|
||||
|
||||
/* "?color=%23eff0f1": the tint, and nothing else so far. */
|
||||
QColor colorOption(const QString &id, QString *name) {
|
||||
QColor color;
|
||||
@@ -85,7 +100,7 @@ QImage ThemeIconProvider::requestImage(const QString &id, QSize *size,
|
||||
if (name.isEmpty())
|
||||
return QImage();
|
||||
|
||||
const QIcon icon = QIcon::fromTheme(name);
|
||||
const QIcon icon = iconForName(name);
|
||||
if (icon.isNull())
|
||||
return QImage(); /* QML shows its fallback glyph */
|
||||
|
||||
@@ -116,8 +131,6 @@ void installThemeIcons(QQmlApplicationEngine *engine) {
|
||||
|
||||
QString iconThemeName() { return QIcon::themeName(); }
|
||||
|
||||
bool themeIconAvailable(const QString &name) {
|
||||
return QIcon::hasThemeIcon(name);
|
||||
}
|
||||
bool themeIconAvailable(const QString &name) { return !iconForName(name).isNull(); }
|
||||
|
||||
} // namespace app
|
||||
|
||||
@@ -10,10 +10,12 @@
|
||||
|
||||
#include <QGuiApplication>
|
||||
#include <QClipboard>
|
||||
#include <QBuffer>
|
||||
#include <QFile>
|
||||
#include <QImage>
|
||||
#include <QJsonDocument>
|
||||
#include <QJsonObject>
|
||||
#include <QPainter>
|
||||
#include <QPointingDevice>
|
||||
#include <QQuickItem>
|
||||
#include <QQmlApplicationEngine>
|
||||
@@ -27,6 +29,8 @@
|
||||
#include <cstring>
|
||||
|
||||
#include "cst_agent.h"
|
||||
#include "cst_attach.h"
|
||||
#include "ui/SessionImages.h"
|
||||
#include "ui/ThemeIcons.h"
|
||||
|
||||
namespace app {
|
||||
@@ -118,7 +122,26 @@ QString writeFixtureSession(const QString &ws, const QString &id) {
|
||||
memset(&m, 0, sizeof(m));
|
||||
m.role = CST_ROLE_USER;
|
||||
m.content = strdup("what is here?");
|
||||
/* The prompt carries a picture, stored exactly the way the engine stores
|
||||
* one (the file next to the session, the message naming it). */
|
||||
{
|
||||
QImage pic(320, 200, QImage::Format_RGB32);
|
||||
pic.fill(Qt::darkCyan);
|
||||
QPainter painter(&pic);
|
||||
painter.fillRect(QRect(0, 0, 160, 100), Qt::darkYellow);
|
||||
painter.end();
|
||||
QByteArray png;
|
||||
QBuffer buf(&png);
|
||||
buf.open(QIODevice::WriteOnly);
|
||||
pic.save(&buf, "PNG");
|
||||
char aerr[256] = "";
|
||||
m.attach = cst_attach_import_bytes(path, "shot.png", png.constData(),
|
||||
static_cast<size_t>(png.size()), aerr,
|
||||
sizeof(aerr));
|
||||
m.nattach = m.attach ? 1 : 0;
|
||||
}
|
||||
cst_session_append(s, &m);
|
||||
cst_attachments_free(m.attach, m.nattach);
|
||||
memset(&m, 0, sizeof(m));
|
||||
m.role = CST_ROLE_ASSISTANT;
|
||||
m.content = strdup("let me look. ");
|
||||
@@ -175,6 +198,12 @@ int runUiTest(AgentCore &core, const QString &size) {
|
||||
|
||||
QQmlApplicationEngine engine;
|
||||
installThemeIcons(&engine); /* the window's icons, same as main() */
|
||||
/* image://cstattach/: the session's stored images, pointed at the session
|
||||
* the engine has open (and re-pointed on every switch). */
|
||||
SessionImages *images = installSessionImages(&engine);
|
||||
images->setDir(core.attachmentDir());
|
||||
QObject::connect(&core, &AgentCore::sessionChanged, images,
|
||||
[images, &core] { images->setDir(core.attachmentDir()); });
|
||||
engine.rootContext()->setContextProperty(QStringLiteral("agent"), &core);
|
||||
engine.rootContext()->setContextProperty(QStringLiteral("initialWidth"), width);
|
||||
engine.rootContext()->setContextProperty(QStringLiteral("initialHeight"), height);
|
||||
@@ -303,6 +332,36 @@ int runUiTest(AgentCore &core, const QString &size) {
|
||||
clickMouse(win, QStringLiteral("sessionRow:") + fixture);
|
||||
check(core.sessionId() == fixture && list && list->property("count").toInt() == 5,
|
||||
QStringLiteral("switching back replays it again"));
|
||||
{
|
||||
bool promptImage = false;
|
||||
const QVariantList rows = core.history();
|
||||
for (const QVariant &v : rows) {
|
||||
const QVariantMap r = v.toMap();
|
||||
if (r.value(QStringLiteral("role")).toString() == QLatin1String("user") &&
|
||||
r.value(QStringLiteral("images")).toString().contains(
|
||||
QLatin1String("|")))
|
||||
promptImage = true;
|
||||
}
|
||||
check(promptImage,
|
||||
QStringLiteral("a stored prompt image comes back with its row"));
|
||||
/* The rows are in the model by now; the cards -- and the thumbnails in
|
||||
* them -- are built on the view's next layout pass. A frame forces it,
|
||||
* and that frame is also the picture a person looks at. */
|
||||
maybeShot(win, QStringLiteral("-images"));
|
||||
QTest::qWait(150);
|
||||
if (qEnvironmentVariableIsSet("CST_APP_DUMP")) {
|
||||
QVariant v;
|
||||
QMetaObject::invokeMethod(win, "rowJson", Q_RETURN_ARG(QVariant, v),
|
||||
Q_ARG(QVariant, 0));
|
||||
std::printf("note: replayed row 0: %s\n", qPrintable(v.toString()));
|
||||
std::printf("note: %d strips and %d thumbs in the tree\n",
|
||||
countNamed(win->contentItem(), QStringLiteral("imageStrip")),
|
||||
countNamed(win->contentItem(), QStringLiteral("imageThumb")));
|
||||
}
|
||||
check(countNamed(win->contentItem(), QStringLiteral("imageThumb")) >= 1,
|
||||
QStringLiteral("the replayed transcript shows it as a thumbnail (%1)")
|
||||
.arg(countNamed(win->contentItem(), QStringLiteral("imageThumb"))));
|
||||
}
|
||||
|
||||
/* --- provider, model, key: the settings dialog ------------------------ */
|
||||
check(core.providers().size() > 0 &&
|
||||
@@ -558,6 +617,72 @@ int runUiTest(AgentCore &core, const QString &size) {
|
||||
check(false, QStringLiteral("assistant deltas are reachable"));
|
||||
}
|
||||
|
||||
/* --- the image provider: only the session's own files ----------------- */
|
||||
{
|
||||
const QString dir = core.attachmentDir();
|
||||
check(!dir.isEmpty() && dir.contains(QLatin1String("/attachments/")),
|
||||
QStringLiteral("the agent names its image directory (%1)").arg(dir));
|
||||
QImage pic(32, 16, QImage::Format_RGB32);
|
||||
pic.fill(Qt::darkGreen);
|
||||
QByteArray png;
|
||||
QBuffer buf(&png);
|
||||
buf.open(QIODevice::WriteOnly);
|
||||
pic.save(&buf, "PNG");
|
||||
char aerr[256] = "";
|
||||
cst_attachment_t *stored = cst_attach_import_bytes(
|
||||
cst_agent_session_path(core.agent()), "probe.png", png.constData(),
|
||||
static_cast<size_t>(png.size()), aerr, sizeof(aerr));
|
||||
check(stored != nullptr,
|
||||
QStringLiteral("a picture can be stored next to the session"));
|
||||
if (stored) {
|
||||
QSize got;
|
||||
const QImage shrunk =
|
||||
images->requestImage(QString::fromUtf8(stored->rel),
|
||||
&got, QSize(8, 8));
|
||||
check(!shrunk.isNull() && shrunk.width() == 8 && shrunk.height() == 4,
|
||||
QStringLiteral("the provider serves it, shrunk to what was asked "
|
||||
"(%1x%2)")
|
||||
.arg(shrunk.width())
|
||||
.arg(shrunk.height()));
|
||||
check(images->requestImage(QStringLiteral("../escape.png"), &got,
|
||||
QSize(64, 64))
|
||||
.isNull(),
|
||||
QStringLiteral("a name outside the attachment directory is refused"));
|
||||
check(images->requestImage(QStringLiteral("no-such-file.png"), &got,
|
||||
QSize(64, 64))
|
||||
.isNull(),
|
||||
QStringLiteral("a file that is not there is refused"));
|
||||
cst_attach_free(stored);
|
||||
}
|
||||
}
|
||||
|
||||
/* --- a tool that made a picture: its card shows it -------------------- */
|
||||
{
|
||||
const int before = list->property("count").toInt();
|
||||
const int thumbsBefore =
|
||||
countNamed(win->contentItem(), QStringLiteral("imageThumb"));
|
||||
event4("engineToolEvent", QStringLiteral("open"), QStringLiteral("call-img"),
|
||||
QStringLiteral("view_image"), QStringLiteral("{\"path\":\"shot.png\"}"));
|
||||
const QString imgs =
|
||||
QStringLiteral("img-fixture.png|shot.png|view_image|24x8");
|
||||
check(QMetaObject::invokeMethod(win, "engineToolImages",
|
||||
Q_ARG(QVariant, QStringLiteral("call-img")),
|
||||
Q_ARG(QVariant, imgs)),
|
||||
QStringLiteral("images a tool made reach the view"));
|
||||
QTest::qWait(150);
|
||||
const QVariantMap card = rowAt(before);
|
||||
check(card.value(QStringLiteral("images"))
|
||||
.toString()
|
||||
.contains(QLatin1String("img-fixture.png")) &&
|
||||
card.value(QStringLiteral("name")).toString() ==
|
||||
QLatin1String("view_image"),
|
||||
QStringLiteral("the card of that call carries the image"));
|
||||
check(countNamed(win->contentItem(), QStringLiteral("imageThumb")) >
|
||||
thumbsBefore,
|
||||
QStringLiteral("its thumbnail is in the card"));
|
||||
maybeShot(win, QStringLiteral("-toolimage"));
|
||||
}
|
||||
|
||||
/* --- a narrow window: dock -> drawer, and the button opens it --- */
|
||||
resizeTo(win, 460, 900);
|
||||
check(docked && !docked->isVisible(),
|
||||
|
||||
Reference in New Issue
Block a user